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0.0  SUMMARY 


Due  to  the  increase  in  the  use  of  Built-In  Test  (BIT)  and  External  Test 
Equipment  (ETE),  for  maintainability  purposes, in  recent  years  the  figures 
of  merit  (FOM),  analyses  techniques  and  demonstration  techniques  currently 
used  for  maintainability  lack  the  capability  of  expressing  the  adequacy  of 
BIT  and/or  ETE  (BIT /ETE).  The  objective  of  this  study  was  to  determine 
what  BIT  /ETE  FOMs  and  their  associated  analysis  /demonstration  techniques 
are  required  to  determine  the  adequacy  of  BIT /ETE.  Also,  how  and  when 
should  these  BIT  /ETE  FOMs  be  specified. 

The  BIT  /ETE  FOMs  defined  and  examined  in  this  study  have  all  appeared 
in  previous  system /equipment  specifications.  However,  few  system  specifica¬ 
tions  have  thoroughly  defined  all  the  BIT  /ETE  objectives  and  in  many  cases 
the  interpretation  of  the  requirements  was  ambiguous.  As  a  result  of  this 
Btudy  a  firm  definition  of  each  of  the  BIT /ETE  FOMs  has  been  established. 
These  definitions  and  the  models  that  define  them  are  summarized  in  Table  1. 
For  the  FOMs  defined  with  "detected  faults",  "detectable  faults"  can  be  in¬ 
terchanged  with  "detected  faults"  without  affecting  the  definition  of  the  FOM. 
For  each  of  the  defined  BIT /ETE  FOMs,  methodologies  have  been  developed 
for  analysis  and  demonstration.  The  analysis  and  demonstration  techniques 
developed  consist  of  existing  techniques ,  modification  of  existing  techniques , 
and  new  techniques.  Table  2  summarizes  the  various  analysis  and  demon¬ 
stration  techniques  that  apply  to  each  BIT /ETE  FOM. 

A  methodology  has  also  been  developed  to  determine  when  each  BIT  /ETE 
FOM  should  be  specified.  The  methodology  correlates  the  various  system/ 
equipment  BIT  /ETE  objectives  with  the  BIT  /ETE  FOMs  that  suit  each  objec¬ 
tive  the  best.  Determination  of  the  most  suitable  FOMs  was  based  on  1)  how 
the  BIT /ETE  FOM  was  related  to  BIT /ETE  objective,  and  2)  how  well  the 
BIT  /ETE  FOM  was  evaluated  as  a  figure  of  merit.  Guidelines  have  also  been 
provided  to  ensure  that  the  specification  of  numerical  values  for  multiple 
FOMs  (i.e.,  when  specifying  related  FOMs  together)  is  consistent  (i.e.,  not 
contradictory) . 

The  final  result  of  this  study  indicates  that  only  minor  alterations  are 
required  to  integrate  BIT  /ETE  FOMs  defined  and  their  corresponding  analysis/ 
demonstration  techniques  into  the  present  maintainability  program  plans.  The 
minor  alterations  that  are  required  have  been  developed  in  this  report. 
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TABLE  2.  SUMMARY  OF  BIT/ETE  FOM  ANALYSIS/DEMONSTRATION  TEfl 


BIT/ETE  FOM 


Analysis  Technique 


Demonstration  Teohnlquo 


•  Fraction  of  Faults  Deteoted  (FFD) 

•  Fraction  of  False  Alarms  (FFA) 

•  Fraction  of  False  .Status  Indications) 
(FFSI) 

•  Mean  Fault  Detection  Time  (TPD) 

e  Mean  BIT/ETE  Running  Time  (TB) 

e  Frequency  of  BIT/ETE  Executions 
<%)• 

e  Test  Thoroughness  (TT) 

e  Fault  Isolation  Resolution  (FIR  (L)) 
e  Fraction  of  Faults  Isolated  (FFI) 
e  Mean  Fault  Isolation  Time  (T^j) 


e  Maintenanc  e  Personnel  Skill  Level 
(MPSL) 


e  RIT/ETE  Reliability  (MTBFB/E) 

e  BIT/ETE  Maintainability 
(MTTRB/E) 

e  BIT/ETE  Availability  <Ab/e> 


e  System  Maintainability  (MTTR) 
e  System  Availability  (A) 


e  Fraotion  of  Erroneous  Fault 
Isolation  Results  (FEFI) 


e  can  be  analyzed  by  a  ratio  of  occurrence  rates 
(e.g. ,  failure  rate) 

e  can  be  analyzed  by  a  ratio  of  ooourrenoe  rates 
(e.g. ,  falluro  rato) 

e  can  be  analyzed  by  a  ratio  of  ooourrenoe  rates 
(e.  g. ,  failure  rate) 

e  can  be  analyzed  by  a  method  similar  to  MIL- 
HDBK-472  procedure  2  or  RADC-TR-78-169, 
a  failure  rate  weighted  average  of  times  (times 
determined  thru  time  line  analysis) 

e  can  be  analyzed  by  time  line  analysis  slnoe 
there  Is  no  randomness  in  its  occurrence 

e  can  be  analyzed  by  time  line  analysis  slnoe 
there  is  no  randomness  in  Its  occurrence 

e  can  be  analyzed  by  a  ratio  of  ooourrenoe  rates 
(e.  g, ,  failure  rate) 

e  can  bo  analyzed  by  a  ratio  of  ooourrenoe  rates 
(e.g, ,  failure  rate) 

e  oan  be  analyzed  by  a  ratio  of  ooourrenoe  rates 
(e.g.,  failure  rate) 

e  oan  be  analyzed  by  a  method  similar  to  MIL- 
HDBK-472,  procedure  2  or  RADC-TR-78-169, 
a  failure  rata  weighted  average  of  times  (times 
determined  thru  time  line  analysis) 

e  oan  be  analyzed  by  a  weighted  average  of  skill 
levels  If  it  is  defined  as  an  average,  otherwise 
It  is  strictly  determined  by  measuring  the  max¬ 
imum  skill  level  required  for  eaoh  maintenance 
action 

e  oan  be  analyzed  using  MIL-IIDBK-217 

e  oan  be  analyzed  using  MIL-HDBK-472, 
RADC-TR-78-169 

e  oan  be  analysed  using  current  techniques  to 
determine  reliability  ar.d  maintainability  (o.  g, , 
MIL-HDBK-217,  MIL-HDBK-472,  etc...) 

e  nan  bo  analyzed  using  MIL-HDBK-472, 
RADC-TR-78-169, 

e  oan  be  analyzed  uelng  ourrent  teohniquee  to 
determine  reliability  and  maintainability  (e.g,, 
MIL-HDBK-817,  MIL-HDBK-472,  eto...) 

e  oan  not  be  analyzed 


can  be  verified  by  a  binominal  diet! 
by  field  data  collection  (FFDu  only: 

can  bo  verified  by  field  data  collect! 


•  can  be  verified  by  field  data  collect 

•  oan  be  verified  by  direot  time  meaai 

i 
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•  oan  be  verified  by  direot  time  meaa,j 

c  ,n  be  verified  by  direot  eime  mess! 

e  can  be  verified  by  direot  meaiurenvj 
the  same  way  FFD  Is  dsmonst rated 
on  how  it  is  defined 

•  oan  be  verifiod  by  a  multinomial  die] 

or  by  field  data  oolleotlon  -■? 

•  oan  be  verified  by  o  binomial  distrli 
by  field  data  collection 

•  can  be  verified  by  teohniquee  slmiii 
MIL-8TD-471  or  by  field  data  colled 


•  oan  be  verified  by  direot  measuring 
field  data  aollectlon 


•  can  be  verified  by  using  the  teohni( 
MIL-8TD-781  or  field  data  oolleotlon 

•  can  be  verified  uelng  the  technique 
MIL-STD-471  or  field  data  oolleotlon 

•  oan  be  verified  by  uelng  the  teohnit 
MIL-8TD-781  end  MIL-STD-471  or  b 
collection 

•  can  be  verified  by  uelng  the  technii 
MIL-STD-471  or  field  data  oollectloq 

s  can  be  verified  by  uelng  the  teohniil 
MIL-STD-781  and  MIL-STD-472  or 
date  collection 

e  oen  be  verified  by  a  binomial  dletrlt 
by  field  data  collection 
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TABLE  2.  SUMMARY  OF  BIT/KTK  FOM  A  NALYSIS/DKMONST  RATION  TECHNIQUES 


_  T'emonatratlon  Technique _ 

•  can  Lie  verified  by  a  binominal  distribution  or 
by  field  data  collection  (FKDjj  only) 

•  can  be  verified  by  field  data  collection  only 

•  can  be  verified  by  field  data  collection  only 

•  can  be  varlfiod  by  direct  time  measurement 


•  aan  be  verified  by  direct  ttmo  measurement 

•  can  be  verified  by  direct  elmo  measurement 

•  can  be  verified  by  dtroct  measurement  or 

the  Barne  way  FKD  is  demonstrated,  depending 
on  how  it  is  defined 

•  can  be  verified  by  n  multinomial  distribution 
or  by  field  data  collection 

•  can  be  verified  by  a  binomial  distribution  or 
by  field  data  collection 

•  can  be  verified  by  techniques  similar  to 
Ml  I.- STD' •  4?  I  or  by  field  data  collection 


•  can  be  verified  by  direct  measurement  or  by 
field  data  collection 


•  can  be  verified  by  using  the  techniques  of 
MIL-STD-781  or  field  data  collection 

•  can  be  verified  using  the  techniques  of 
MII.-STD-4TI  or  field  data  collection 

•  can  be  verified  by  using  the  techniques  of 
MIL-STD-781  and  MU, -STD-471  or  by  field  data 
collection 

•  can  bo  verified  by  using  the  techniques  of 
M1L-STD-471  or  field  data  collection 

•  can  be  verified  by  uelng  the  techniques  of 
MIL-STD-781  and  MlL-STD-472  or  by  field 
dsts  collection 

•  can  be  verified  by  a  binomial  distribution  or 
by  field  data  collection 
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EVALUATION 


The  objective  of  this  study  was  to  Investigate  and  determine  figures  of 
merit  that  could  be  used  In  specifications  as  requirements  for  Built-in-Test 
(BIT)  and  external  tester  adequacy  as  wel)  as  the  corresponding  demonstration 
techniques  and  procedures  that  could  oe  used  to  verify  that  the  figures  of 
merit  have  been  achieved. 

The  objectives  have  been  satisfactorily  fulfilled.  The  final  report 
provides  the  Information  necessary  to  adequately  specify,  analyze,  and 
demonstrate  the  BIT/ETE  capabilities  contained  In  a  system/equipment.  The 
methodologies  presented  are  compatible  with  existing  maintainability  program 
elements  and  allow  BIT/Elt  requirements  to  be  easily  Integrated  Into  standard 
maintainability  programs. 

The  use  of  the  results  of  this  effort  provides  the  foundation  for  the 
consistent  specification  and  verification  of  effective  BIT/ETE  figures  of 
merit  In  electronic  equlpment/system  acquisitions. 

JERRY  F.  LIPA,  JR. 

Project  Engineer 
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1.0  INTRODUCTION 

This  dooument  presents  the  results  of  a  study  to  investigate  and  determine 
the  measures  and  figures  of  merit  that  should  be  used  in  specifications  as  require 
ments  for  Built-in  Test  (BIT)  and  External  Test  Equipment  (ETE)  adequacy.  This 
study  was  performed  under  Contraot  F30602-78-C-0137  with  Rome  Air  Develop¬ 
ment  Center.  This  report  is  prepared  in  aocordanue  with  CDRL  Item  A002  and 
data  item  description  DI-S-3591A/M. 
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1.1  DEFINITION  OF  THE  PROBLEM 

In  recent  years  the  uso  of  Built-in  Tost  (BIT!  and/or  External  Teat  Equip¬ 
ment  (ETE)  as  maintenance  tools  has  increased  significantly .  For  purposes  of 
this  study  BIT  and  ETE  are  defined  ast 

Built-In  Teat  (BIT)  -  That  capability  internal  to  an  oquipment/aystom  which 
is  provided  for  the  purpose  of  failure  detection  and/or  isolation.  Includes 
built-in  test  equipment  (BITE),  software  programs,  firmware  programs, 
tost  circuitry,  maintenance  panels,  status  indicators,  etc. 
External-Test-Equipment  (ETE)  -  That  special  purpose  or  general  purpose 
test  equipment  external  to  an  equip ment/sys tom  which  Is  designated  for 
uso  In  the  failure  detection  and/or  isolation  process. 

The  figures  of  merit,  analysis  techniques,  and  demonstration  techniques 
that  are  currently  used  for  the  purpose  of  maintainability  lack  the  capability  of 
expressing  the  adequacy  of  BIT  and/or  ETE  (BIT/ETK)  within  a  system/ 
equipment,  The  objective  of  this  study  was  to  determine  the  measures  and  fig¬ 
ures  of  merit  that  are  required  to  determine  BIT/ETK  adequacy.  Furthermore, 
methodologies  were  to  be  developed  to  analyse  and  demonstrate  these  measures. 
Specific  objectives  included! 

1)  survey  current  figures  of  merit  to  determine  their  usefulness  and 
completeness  for  BIT/ETK  specification  and  determine  other  figures 
of  merit  required,  appropriate  to  BIT/ETK  specification, 

2)  determine  methods  of  measurement  and  demonstration  for  the  associ¬ 
ated  figures  of  merit. 

3)  provide  guidance  for  the  specification  of  appropriate  figures  of  merits 
and  their  numerical  values  in  maintainability  requirements. 

i)  provide  guidance  pertaining  to  the  integration  of  BIT/ETK  requirements 
and  analysis/demonstration  techniques  into  the  current  maintainability 
program  plans. 


1.2  APPROACH 

The  approach  to  satisfying  the  study  objectives  consisted  of  the  following? 

1)  Data  collection  -  This  task  consisted  of  surveying  the  BIT/ETE  FOMs 
that  are  currently  used  and  identifying  what  methodologies  exist  for 
analysing  and  demonstrating  those  FOMs.  The  outcome  of  this  task 
was  used  to  aid  in  the  identification  of  additional  BIT/ETE  FOMs  or 
analysis  /demonstration  methodologies  required, 

2)  FOM  evaluation  -  This  task  consisted  of  evaluating  tho  suitability  of 

the  defined  FOMs  as  design  specifications.  A  weighted  rating  evaluation 
approach  was  used. 

3)  AnalyslB/demonstration  techniques  development  -  This  step  consisted 
of  developing  appropriate  analysis  and  demonstration  techniques  for  the 
defined  FOMb.  The  resulting  techniques  are  a  combination  of  existing, 
modified  and  new  methodologies. 

4)  FOM  specification  guidelines  -  This  task  consisted  of  developing  u 
procedure  for  determining  what  BIT/ETE  FOMs  should  be  specified  for 
given  systom/oquipment  objectives. 

5)  integration  of  BIT/ETE  FOMs  into  maintainability  programs  -  This  task 
consisted  of  determining  how  the  newly  developed  BIT/ETE  FOMs  and 
their  associated  unalyBis/domonBtration  techniques  should  be  implemented 
into  a  maintainability  program  plan. 


2.0  DATA  COMJCTQN 


2.1  OBJECTIVE /APPROACH 

A  data  collection  effort  was  undertaken  to  determine  what  BIT/ETE  FOMs  are 
currently  being  used,  what  current  methodologies  are  being  used  for  analysing 
and  testing  BIT/ETE>,  what  are  the  Accepted  quantitative  and  qualitative  definitions 
for  typical  BIT/ETE  FOMs,  and  what  inherent  fault  detection  or  fault  isolation 
characteristics  are  not  Adequately  covered  by  the  current  BIT/ETE  FOMs.  The 
data  collection  effort  consisted  of  three  separate  tasks t 

.1.  literature  search, 

2.  system  specifications  search,  and 

3.  Industry  survey 

The  following  subsections  summarise  the  approach  and  findings  of  each  data 
collection  task. 
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2.2  LITERATURE  SEARCH 

The  first  task  performed  In  the  data  collection  effort  was  on  extensive  litera¬ 
ture  searoh.  The  literature  search  consisted  of  two  different  efforts;  1)  Burvey 
of  military  standards  and  handbooks  for  ourrent  BIT/ETE  requirements  and  de¬ 
finitions  and,  2)  survey  of  technical  publications  related  to  BIT/ETE. 

2.2.1  Specification  Review 

Table  3  is  a  list  of  the  documents  examined  that  relate  to  BIT/ETE  require¬ 
ments.  Investigation  of  the  table  Indicates  that  a  majority  of  the  specifications 
reference  MIL-STD-415  when  specifying  requirements  on  BIT/ETE.  MIL-STD- 
410,  paragraph  5.2.3,  whloh  is  reproduced  in  Figure  1,  only  contains  qualitative 
requirements  on  BIT/ETE.  The  only  specification  that  contained  any  quantitative 
requirements  on  BIT/ETE  waB  NAVAIR  AR-10.  NAVAIR  AR-10  contained  speci¬ 
fic  requirements  on  the  proportion  of  faults  dotooted/lsolated  by  BIT/ETE,  and 
the  fault  Isolation  ambiguity  level.  However,  NAVAIR  AR-.10  Is  only  applicable 
to  NAVAIR  avlonlo  equipment. 

The  only  other  dooumont  that  contains  Information  on  the  specification  of 
BIT/ETE  requirements  is  NAVMAT-3960.  NAVMAT  3960  provides  guidelines 
for  the  design  and  specification  of  BIT/ETE,  but  does  not  attempt  to  relate  these 
requirements  to  quantitative  FOMs.  The  FOMs  which  are  discussed  Include! 

•  Aw  liability 

•  Reliability 

•  Mean  Corrective  Maintenance  Tlmo 

•  Fault  Definition 

•  BIT  Detectability  Lovel 

c  BIT  Fault-Isolation  Level 

•  BIT  False-Alarm  Rate 

•  BIT  Self-Test  Requirement 

•  Extent  of  Operator  Participation 

•  Software  Constraints  (memory  capacity) 

•  Design  Growth  Limits 

•  Design  Cost  Goal  (contract  specification) 

•  BIT  Fail-Safe  Provisions 

•  Fault  Indicators 

•  Special  BIT  Features 

•  BIT  Calibration  Requirements 
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TABLE  3.  SUMMARY  OF  BIT/ETE  REQUIREMENTS  CURRENTLY  SPECIFIED  IN  MILITARY  STANDARDS 
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6.2,3  BIT_c«jj»billty,  -  Tha  built-in-tait  (BIT)  capability  mail  l>a  mcorpof  *t*d  at  raqulrad  by  tha  contract  to 
mure  alfective  Implementation  of  the  defined  maintenance  concept.  The  built-in-teat  capability  shell  coniitt  of 
the  following; 

(a)  Self-test  provlilont:  Self-tett  ptovlilont  thall  be  an  inherent  part  of  an  item.  Thete  provitlont 
thall  terve  a  dual  function;  Itonr  performance  evaluation,  end  complementing  BIT  provlilont  to 
provide  item  totting.  When  talf-tait  provlilont  ara  practical,  the  contractor  thall  ute  them; 
however,  their  ute  thall  not  jeopardize  the  operation  or  performance  of  the  Item. 

(b)  Marginal  tatting;  When  critical  Item  peremeteri  or  charecterlitlct  ere  subject  to  change  or  drift 
and  BIT  capabllitlet  mutt  be  uted,  thete  areet  thall  be  tatted  by  marginal  tatting  technlquat  at 
defined  in  thlt  ttandard. 

0.2.3. 1  Applicability  of  toit  provltlon  clatiet,  -  Clan  A  and  B  test  provlilont  thall  be  applicable  to  tha  BIT 
capability. 

B.2.3.2  BIT  provitlont,  -  BIT  provitlont  thall  ba  added  to  an  Item  for  tha  tola  pur  pom  of  tatting  tha  Item. 
They  thall  be  ilmple  In  detign  and  operation,  accurate,  eatlly  maintained,  preferably  more  reliable  than  the 
circuitry  providing  performance,  and  thall  not  degrade  the  performance  of  the  Item  In1  which  they  are  Incorporated, 

®'2.3>3  Saw  of.  operation.  -  BIT  provitlont  thall  provldo  optimum  convenience  of  u»e  end  operation.  The 
detign  of  control!  and  reed-out  devlcet  thall  be  tuch  that  they  can  ba  aatily  uted  and  Interpreted  by  low  tkill 
per  tonne!,  To  the  maximum  extent  It  thall  be  poulble  to  operate  the  provitlont  with  minimum  reference  to  Item 
handbooks  The  need  for  external  equipment  or  toolt  to  tupplement  thlt  teitlng  capability  thall  ba  minimized. 


Figure  1,  BIT/ETE  Requirements  of  MIUHTD-415 

v- 


\ 

J 


If 


i 


-I 

» 

\ 


\ 


p' 

V; 

i 

v 

L 


if 

i 


i,  | 

ti' 


p 

rt 

f 


2.2.2  Toohnloal  Publications  Review 

Sources  for  tho  technical  publications  literature  search  were; 

1.  Defense  Documentation  Center  (DDC) 

2.  NASA  Scientific  and  Technical  Information  Division 

3.  Hughes  Aircraft  Technical  Library 

A  majority  of  tho  publications  reviewed  dealt  with  the  design  of  BIT/ETE,  but 
did  not  address  BIT/ETE  specification,  analysis,  or  demonstration.  The  only 
dooumont  that  contained  any  Information  about  the  specification  of  DIT/ETE  was 
A  Guido  to  the  Application  of  Built-In  Test  to  Navy  Avionic  Equipment,  by  ARINC 
Research  Corporation  (AD  837  094),  One  section  was  devoted  to  the  specification 
of  BIT/ETE,  however,  tho  content  of  that  section  merely  stressed  the  need  for 
better  ways  to  specify  BIT/ETE.  It  also  gave  relative  guidelines  on  what  the 
specifications  should  be  capable  of  (l.e,,  BIT/ETE  specifications  should  be 
capable  of  demonstration). 
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2.3  SYSTEM  SPECIFICATION  SURVEY 

The  second  data  collection  task  used  to  Identify  the  BIT/ETE  FOMs  that  are 
currently  In  use  was  to  survey  a  representative  sample  of  system  specifications. 
The  review  Inoluded  specifications  for  forty-nine  systems  In  which  Hughes  has 
been  cither  the  contractor  for,  or  a  candidate  contractor  for.  These  forty-nine 
system  specifications  oover  a  broad  range  of  environments  with  1 1  airborne, 

16  ground,  18  shipboard,  4  missile,  and  1  space  (one  system  Is  used  In  ground 
and  airborne  environments), 

A  review  of  the  composite  collection  of  FOMs  Identified  In  the  system  speci¬ 
fication  review  indicates  that  all  tho  FOMs  fall  into  seventeen  generic  groupings. 
The  specific  FOMs  within  each  group  vnry  In  numerical  value  and  exact  definition 
but  all  relate  to  the  same  generic  fault  detection  and/or  fault  Isolation  character¬ 
istic.  The  general  BIT/ETE  FOMs  Identified  and  their  various  forms  encountered 
were! 

1)  fraction  of  faults  detected! 

e  percent  of  all  faults  au'owi  tloally  detected  by  BIT/ETE 

•  percent  of  all  faults  detectable  by  BIT/ETE 

e  percent  of  all  faults  detectable  on-line  by  BIT/ETE 

•  percent  of  all  faults  and  out -of -tolerance  conditions  detectable  by 
BIT/ETE 

•  percent  of  all  faults  detectable  by  any  means 

2)  fraction  of  false  alarms 

•  rate  at  which  false  indications  occur  (per  10^  hours) 

•  percent  of  Indicated  failures  oaused  by  actual  failures 

•  percent  of  BIT/ETE  Indicated  failures  caused  by  actual  failures 

•  peroent  of  BIT/ETE  fault  Isolations  to  the  wrong  I  JUT 

3)  fraction  of  false  status  IndloatlonB 

•  peroent  of  erroneous  BIT  Indications 

4)  mean  fault  detection  time 

•  time  to  Indicate  a  fault  once  it  has  occurred 

•  time  to  detect  a  fault  once  It  has  oocurred 

8)  mean  BIT/ETE  running  time 

•  time  to  verify  that  a  failure  has  occurred/or  has  been  repaired 
using  BIT/ETE 

6)  frequency  of  BIT/ETE  executions 

•  time  Interval  between  BIT/ETE  executions 
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test  thoroughness 

•  percent  of  all  equipment  functions  tested 
fault  Isolation  resolution 

•  Isolation  of  Pj  percent  of  the  failures  to  Xi  LRUs,  P2  Pero<>nt  of  the 
failures  to  Xo  LRUs  and  so  on,  with  any  fault  Isolation  method. 

•  Isolation  of  all  faults  to  less  than  or  equal  to  some  maximum  number 
of  LRUs. 

•  Isolation  of  Pj  percent  of  the  failures  to  X|  LRUs,  Pa  peroent  of  the 
failures  to  Xa  LRUs,  aixl  so  on,  with  BJT/ETE 

•  isolation  of  a  specified  percent  of  the  failures  to  less  than  or  equal 
to  a  specified  quantity  of  LRUa  at  the  various  maintenance  levels. 

•  Isolation  of  a  specified  percent  of  the  failures  down  to  less  than  or 
equal  to  a  maximum  number  of  plug-in  modules. 

•  Isolation  semi-nutomatically  to  a  certain  percent  of  all  faults  down 
to  a  specified  number  of  LRUs. 

fraction  of  faults  isolated 

•  isolate  a  certain  peroent  of  all  failures  that  occur 

•  isolate  a  certain  poi'oont  of  all  failures  that  occur  with  IUT/ETK 
mean  fault  isolation  time 

•  Isolate  a  specif  led  percent  of  fatlures  that  occur  within  a  specified 
maximum  time, 

•  Isolate  a  failure  down  to  a  replaceable  level,  within  a  specified 
average  time. 

•  isolate  a  failure  down  to  a  replaceable  level  wtthln  a  specified  time 
once  the  fault  Isolation  process  hns  been  Initiated. 

maintenance  personnel  skill  level 

•  all  maintenance  actions  must  be  capable  of  being  performed  by  a 
specified  quantity  of  maintenance  personnel  with  a  specified  skill 
level,  at  various  maintenance  levels. 

•  BtT/KTK  must  be  designed  for  use  by  a  specified  minimum  skill 
level  technician 

BIT /KT K  mean-i  uno-to-eopatr 
s  menn-tlme-to -repair  ETE 

•  mean-tlme-to-repatr  monltorlng/fault  Isolation  (Unctions 


13)  BIT/ETE  roean  time  between  failures 

•  mean  time  between  failures  of  monltorlng/fault  Isolation  funottons 

•  mean  time  between  failures  of  ETE  only 

14)  BIT/ETE  availability 

e  monltorlng/fault  isolation  functions  should  be  operating  with  a 
specified  probability  of  survival. 

18)  mean-tlme-to-ropalr 

e  system/equipment  MTTR  6  maximum  repair  time 

•  system/equipment  MTTR  &  maximum  repair  time  at  various 

maintenance  levels  \ 

16)  availability  \ 

e  Inherent  availability  \ 

e  operational  availability 

17)  active  memory  allocated  for  BIT/ETS  functions 

e  monltorlng/fault  Isolation  functions  shall  take  up  a  speoifled  peroent 
of  active  oomputer  memory. 

A  summary  of  the  system  specifications  reviewed  aiid  BIT/ETE  FOMs 
identified  in  each  Is  shown  in  Table  4. 
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TABLE  4.  SUMMARY  OF  BIT/ETE  FOMb  SPECIFIED  IN  SYSTEM  REQUIREMENTS 


TABLE  4.  SUMMARY  OF  BIT/ETE  FOMs  SPECIFIED 
IN  SYSTEM  REQUIREMENTS  (Continued) 


2.4  INDUSTRY  SURVEY 

The  third  task  of  the  data  collection  effort  was  an  Industry  survey.  The  ob¬ 
jective  of  this  task  was  to  broaden  the  data  base  obtained  through  the  literature 
search,  and  the  system  specification  survey,  by  surveying  reliability/ 
maintainability  engineers  from  companies  that  are  familiar  with  the  use  of  RMA 
techniques  and  the  speclflcatlon/test  of  B1T/ETE  requirements  In  DoD  contracts. 

A  list  of  candidate  companies  was  extracted  from  the  list  of  Government 
Industry  Data  Exohange  Program  (GIDEP)  subscribers.  A  BIT/ETE  survey  ques¬ 
tionnaire  was  submitted  to  the  reliability /maintainability  engineers  of  each  re¬ 
spective  company.  Figure  2  is  a  oopy  of  the  questionnaire  form  submitted.  The 
questionnaire  was  set  up  to  collect  the  following  information: 

1)  BIT/ETE  FOMs  that  are  currently  used  and  type  of  system  they  are 
used  on 

2)  recommendations  for  new  BIT/ETE  FOMs 

3)  user  critique  of  the  BIT/ETE  FOMs  Identified. 

Of  the  one  hundred  thirty-nine  questionnaires  sent  out,  thirty-one  responses 
were  received,  for  a  22  peroent  return.  Of  the  thirty-one  responses  received, 
twenty-eight  responses  contained  usable  data.  The  remaining  responses  wer«* 
blank. 

Part  I  of  the  BIT/ETE  questionnaire  (Figure  2)  was  uBed  to  Identify  what  BIT/ 
ETE  FOMs  eaoh  respective  engineer  has  had  experience  with  (either  in  analysis, 
demonstration,  or  specification)  and  the  types  of  equipments  (e.g.,  shipboard, 
ground,  airborne,  eto. . . )  they  have  been  used  on.  In  order  to  aid  the  engineers 
that  were  polled,  the  BIT/ETE  FOMs  Identified  to  date  were  tabulated  for  their 
oonvenlenoe.  Space  was  provided  for  additional  FOMs.  The  results  of  this  part 
of  the  survey  Indicate  that  the  BIT/ETE  FOMs  listed  are  FOMs  that  have  been 
encountered  in  one  way  or  another  (1.  e. ,  in  specification,  analysis,  or  demon¬ 
stration)  for  the  various  system  types.  The  results  also  showed,  as  expeoted, 
the  laok  of  experience  in  demonstrating  several  of  the  BIT/ETE  FOMs  tabulated 
such  as;  false  alarm  rate/false  status  Indications,  fault  Isolation  ambiguity  level, 
and  peroent  of  fault  isolation  with  BIT/ETE. 

Part  n  of  the  BIT/ETE  questionnaire  (Figure  2)  was  used  to  obtain  eaoh 
engineer's  view  of  the  usefulness  of  the  BIT/ETE  FOMs  Identified.  This  part 
of  the  questionnaire  consisted  of  a  scoring  oheokllBt  to  rate  eaoh  FOM  according 
to  the  following  suitability  factors:  translatablllty,  traokablllty,  demonstrablllty, 


OTWCR: 


ambiguity,  generality,  and  ooat.  The  results  obtained  through  this  part  of  the 
survey  consisted  of  soores  (ranging  from  1  to  10,  with  10  being  the  best  possible 
soore)  for  eaoh  FOMs  suitability  faotors.  Since  the  soorlng  oheokllsts  results 
did  not  show  very  high  oorrelation  (1.  e, ,  a  majority  of  the  soores  range  from  1  to 
10  for  eaoh  FOM)  the  results  will  be  used  only  as  a  guideline  In  the  assignment  of 
soores  In  the  BIT/ETE  Evaluation  Task  (Section  4).  A  summary  of  the  average 
soores  obtained  for  eaoh  BIT/ETE  FOM  is  provided  In  Tables  5  through  8.  The 
raw  data  Is  also  provided  in  Appendix  C. 
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Scitabaity  Factor 


TABLE  7.  BIT/ETE  INDUSTRY  SURVEY  SUMMARY 


TABLE  8.  BTT/ETE  INDUSTRY  SURVEY  S 


TABLE  9.  BIT/ETE  INDUSTRY  SURVEY 
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2.5  DATA  COLLECTION  SUMMARY  ] 

The  data  collection  tasks  (literature  search,  system  specification  survey, 
and  Industry  survey)  have  resulted  In  the  Identification  of  eighteen  generic  BIT/ 

ETE  FOMs  that  have  been  used  in  prior  system  specifications.  These  BIT/ETE  j 

FOM  types  are; 

1)  fraction  of  faults  detected  (FFD)  | 

2)  fraction  of  false  alarms  <FFA)  | 

3)  fraction  of  false  status  Indications  (FF8I) 

4)  mean  fault  detection  time  (TFD)  ; 

5)  mean  BIT  /ETE  running  time  (Tg) 

6)  frequency  of  BIT/ETE  executions  (Fg)  :] 

7)  test  thoroughness  (TT) 

8)  fault  Isolation  resolution  (FIR(L))  { 

9)  fraction  of  faults  isolated  (FFI)  ;; 

10)  mean  fault  Isolation  time  (Tpj) 

11)  maintenance  personnel  skill  level  (MP8L) 

18)  BIT/ETE  maintainability  (MTTRB/E) 

18)  BIT/ETE  reliability  <MTBFB/E) 

14)  BIT/ETE  availability  (A*  >«) 

15)  MTTR 
18)  A 

17)  memory  allocated  for  BIT/ETE  (FMAB) 

18)  physical  oharaot eristic  FOMs  (c.g, ,  weight,  cost,  etc.,.) 

Currently  there  are  no  standardised  techniques  available  for  analysing  or 

demonstrating  BIT/ETE  FOMs.  The  only  techniques  that  exist  are  maintainability 
analysts  and  demonstration  techniques  (e.g.,  MIL-HDBK-472,  MIL-STD-471, 

RADC-TR-70-88,  RADC  TR-78-189,  eto, , . ),  However  these  techniques  can  only 

be  used  to  measure  a  few  of  the  FOMs  identified  above.  The  remainder  of  this  1 

■  .  .. 

report  concentrates  on  the  development  of  analysis/demonstration  techniques  for  the  < 

most  suitable  FOMs  that  are  listed,  • 

l 

i 
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3.0  B1T/ETE  FQM  IPENTIFICATIQN 

Thin  section  examines  the  basic  objectives  of  specifying  BIT/ET1J  FOM  re¬ 
quirements,  reviews  and  uniquely  defines  the  currently  used  FOMs  identified  in 
Section  8,  correlates  the  ourrent  FOMs  with  the  defined  objectives,  and  identifies 
new  FOMs  required  to  adequately  address  all  BIT/ETE  objectives. 


i 


/ 

/ 


3. 1  IDENTIFICATION  OF  THE  BIT/ETE  OBJECTIVES 

The  objectives  of  BIT/ETE  can  be  broken  down  into  two  distinct  categories 
as  shown  in  Figure  3.  The  first  category  considers  the  characteristics  of  the 
BIT/ETE  itself,  and  the  second  category  considers  the  actual  performance  cap¬ 
abilities  of  the  BIT/ETE. 

The  characteristics  of  BIT/ETE  can  be  further  broken  down  into  the  physical 
characteristics  of  BIT/ETE  (e.g,,  weight,  volume,  component  count),  and  the 
operational  characteristics  of  BIT/ETE  (d.g, ,  BIT/ETE  reliability,  maintain¬ 
ability).  /it  should  be  noted  that  while  BIT/ETE  characteristics  do  not  directly 
relate  to  the  performance  capability  of  BIT/ETE,  generally  the  larger  the  physi¬ 
cal  attributes  of  the  BIT/ETE  the  greater  the  BIT/ETE  capability.  Likewise  the 
logoi'  the  BIT/ETE  reliability  is,  the  greater  the  complexity  and  correspond¬ 
ingly  the  greater  the  capability. 

The  BIT/ETE  curability  is  further  subdivided  into  the  fault  detection  capa¬ 
bility  of  the  BIT/ETE,  and  the  fault  Isolation  capability  of  the  BIT/ETE,  V  ithin 
these  subdivisions  the  BIT/ETE  capability  is  broken  down  into  three  main 
objectives! 

1)  how  much  time  it  takes  to  detect  n  fault  (or  isolate  u  fault) 

2)  how  thorough  is  the  BIT/ETE  fault  detection  (or  fault  isolation)  function 

3)  how  accurate  is  the  BIT/ETE  fault  detection  (or  fault  isolation)  function. 

Since  the  primary  objectives  of  specifying  BIT/ETE  FOMb  relate  to  perform¬ 
ance  capabilities  or  operational  considerations,  the  remainder  of  this  report  will 
concentrate  in  these  areas.  The  BIT/ETE  physical  characteristics  are  straight 
forward  and  require  no  unique  methodology  for  analysis  or  demonstration. 
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3.2  DEFINITION  OF  IDENTIFIED  BIT/ETE  FOMs 

Of  the  eighteen  general  BIT/ETE  FOM  types  identified  thru  the  data  col¬ 
lection  task,  sixteen  of  them  pertain  to  BIT/ETE  capabilities  and  operational 
characteristics.  The  remaining  two  generic  FOMs  (memory  allocated  for  BIT/ 
ETE  and  physical  characteristic  FOMs),  are  related  to  the  physical  charao- 
istics  of  BIT/ETE  and  Will  not  be  further  analyzed  or  discussed  in  the  suc¬ 
ceeding  sections.  Table  10  contains  a  summary  of  the  sixteen  BIT/ETE  FOMs 
(i.e.,  FOMs  related  to  BIT/ETE  capabilities  and  operational  characteristics), 
their  definitions,  and  the  general  model  for  determining  them. 

For  the  FOMs  defined  with  "detected  faults",  "detectable  faults"  oan  be 
interchanged  with  "detected  faults"  and  the  FOM  definition  will  still  be  valid. 
For  the  purposes  of  this  study  "detectable  faults"  refers  to  faults  that  can 
be  detected  when  evaluating  a  FOM  by  analysis,  and  "deteoted  faults"  refers 
to  faults  that  are  detected  when  evaluating  a  FOM  by  a  formal  demonstration. 
For  simplification,  only  "deteoted  faults"  will  be  addressed  for  the  remainder 
of  this  report . 

It  should  be  noted ,  that  the  models  presented  in  this  section  are  of  gen¬ 
eral  forms.  Spedfia  models  to  quantify  each  FOM,  either  by  analysis  or 
demonstrating  are  presented  in  later  sections. 

3,2,1  Definition  of  Fault 

The  definition  of  most  BIT  /ETE  FOMs  includes  a  reference  to  faults  or 
failures.  The  definition  of  these  terms  oan  significantly  affect  the  BIT/ETE 
FOM  meaning  and  muBt  be  clearly  understood.  As  defined  in  MIL-STD-721 
and  MIL-STD-130S,  a  failure  is  defined  as: 

The  inability  of  an  item  to  perform  within  previously  specified  limits ,  or 
a  malfunction  that  oauses  degradation  or  complete  loss  of  equipment 
performance. 

As  defined  in  MIL-STD-1309,  a  fault  is  defined  as: 

A  degradation  in  performance  due  to  detuning,  maladjustment ,  misalign¬ 
ment,  failure  of  partB,  and  so  forth. 

In  general  application,  faults  typically  include  any  hardware  abnormality 
whereas  failures  only  include  those  faults  which  affect  equipment  (subsystem, 
system,  eteo.)  performance  or  mission  accomplishment.  In  practice,  the  appli¬ 
cation  of  a  failure  /fault  definition  la  equipment  or  mission  related  and  should 
be  defined  in  the  equipment  specification,  For  purposes  of  this  report  all 
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BIT /ETE  FOM  definitions  reference  faults  as  opposed  to  failures,  The  defin¬ 
itions  ore  not  restricted  however  and  can  apply  to  failures  as  well,  if  so  de¬ 
sired,  and  if  properly  defined  in  the  equipment  specification, 

3.2.2  Fraction  of  Faults  Detected  (F FD ) 

The  fraction  of  faults  detected  by  BIT/ETE  (FFD)  cun  be  expressed  two 
ways:  1)  the  fraction  of  all  faults  detected  by  BIT/ETE  (FFDA) ,  and  2) 
the  fraction  of  all  detected  faults  detected  by  BIT/ETE  (FFDD). 

FFD  a  is  defined  ns  the  fraction  of  all  faults  that  can  occur,  which  arc 
detected  by  the  BIT  (and/or  ETE)  funation.  ThiB  is  represented  by  the 
following  model: 


FFD  a  quantity  of  faults  detected  by  BIT/ETE  (Qbdf) 

A  quantity  of  all  faults  (Q^) 

l'FD„  is  defined  as  the  fraction  of  fuults  that  can  be  dejected,  which  arc 
detected  by  the  BIT  (and/or  ETE)  function.  This  is  represented 
by  the  following  model: 

rt?n  _  quantity  of  faults  detected  by  BIT/ETE  (qdf) 

D  "  - quantity  of'TauTfilTeTeoTe’a”^^ - 

For  the  above  definitions  the  following  ground  rules  have  boon  established: 

1)  the  quantity  of  all  faults  (Qp),  the  quality  of  faults  detected  by 
BIT/ETE  (Qbdf)»  «nd  the  quantity  of  faults  detected  (Qpn)  exclude 
the  ooourrenoo  of  false  alax*mw  (false  alarms  are  defined  in  Section 

3.2.3). 

2)  intermittent  faults  are  classified  as  a  single  fault,  thuo  Qp,  Qpp, 
and  QnD  include  the  occurrence  of  intermittent  faults  only  once. 

3)  temporary  faults  (faults  caused  by  external  transients  or  noise) 
are  not  classified  as  faults,  therefore  they  arc  excluded  from  Q^, 
QpD ,  und  Qqpd  . 


3.2.3  Fraction  of  Paige  Alarms  (FFA) 

Tho  Fraction  of  False  Alarms  (FFA)  Is  defined  as  the  fraction  of  all  BIT/ETE 
fault  indications  which  are  false  alarms.  False  alarms  are  those  Indications  of 
a  fault  when  an  actual  fault  has  not  occurred.  FFA  Is  represented  by  the  follow¬ 
ing  model) 

quantity  of  BIT/ETE  false  alarms  (QFA) 
quantity  of  all  B1T/ETE  Indicated  failures 

False  alarms  are  dependent  upon  several  factors.  First  of  all,  false  alarms 
are  dependent  upon  tho  BIT/ETE  philosophy.  If  BIT/ETE  1b  considered  an  inte¬ 
gral  part  of  the  system  (1.  e. ,  BIT/ETE  are  considered  part  of  the  system/ 
equipment),  then  Table  11  below  summarises  when  a  false  alarm  will  occur. 


TABLE  11.  OCCURRENCE  OF  FALSE  ALARMS  WITH  INTEGRAL  BIT/ETE 


Status  of 
the 

BIT/ETE 

Equipment 

Status 

Indlonted  by 
BIT/ETE 

BIT/ETE  Indication  Status  Relative 
to  the  Actual  Equipment  Status 

Equipment  Is  UP 

Equipment  is  DOWN 

UP 

up 

OK  -  operational 

undetected  fault 

up 

down 

false  alarm 

OK  -  detected  fault 

down 

up 

undetected  fault 

undetected  fault 

down 

down 

OK  -  detected  fault 

OK  -  detected  fault  (dual) 

If  BIT/ETE  Is  Independent  of  the  system  (1.  e. ,  BIT/ETE  faults  are  not  consid¬ 
ered  system/equipment  faults),  then  Table  12  below  summarises  when  a  false 
alarm  will  occur. 


TABLE  12.  OCCURRENCE  OF  FALSE  ALARMS  WITH  INDEPENDENT  BIT/ETE 


Status  of 
the 

BIT/ETE 

Equipment 
Status 
Indicated  by 
BIT/ETE 

BIT/ETE  Indlcat 
to  the  Actual  E 

fon  Status  Relative 
qutpment  Status 

Equipment  is  UP 

Equipment  Is  DOWN 

up 

up 

OK  -  operational 

undetected  fault 

up 

down 

false  alarm 

OK  -  detected  fault 

down 

up 

OK  -  operational 

undetected  fault 

down 

down 

false  alarm 

OK  -  detected  fault 
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False  alarms  are  categorized  into  three  types  as  follows) 

1)  Faulty  BIT/ETE  function  -  If  a  BIT/ETE  fault  occurs  whloh  Indicates 
the  equipment  is  down  when  It  Is  actually  operational  then  a  false  alarm 
has  occurred. 

2)  out-of-tolerance  conditions  -  False  alarms  oaused  by  out-of-toleranoe 
conditions  occur  when  the  BIT/ETE  measured  an  Internal  signal  and 
determines  It  to  be  but  of  tolerance,  when  the  actual  output  signal  (the 
signal  of  importance)  is.  still  within  its  specified  tolerance  bounds. 

These  types  of  false  alarms  are  largely  dependent  upon  the  circuit  and/ 
or  BIT/ETE  designer (s)  who  set  the  tolerance  bounds  for  each  signal. 

3)  transient  conditions  -  Fault  Indications  caused  by  transient  conditions 
oan  be  classified  as  false  alarms  if  the  transient  does  not  result  In  a 
true  fault  condition. 

There  are  several  conditions  that  may  be  thought  of  as  false  alarms  when  aotual 
faults  have  occurred.  Fault  conditions  that  are  not  false  alarms  lnoludet 

s  intermlttcnts  -  faults  that  exist  only  temporarily  (e.  g. ,  a  fault  indicated 
in  an  airborne  environment  that  can  not  be  recreated  (or  verified)  on  the 
ground). 

e  transients  that  result  In  temporary  faults  1,  c. ,  the  transient  results  in 
a  fault  condition  and  as  a  result,  also  causes  a  temporary  failure  In  the 
sy  stem/equipment. 


3,2.4  Fraction  of  False  Status  Indication  a  (FFSI) 

The  fraotion  of  false  status  Indications  (FFSI)  Is  an  extension  of  FFA.  FFS1 
Is  defined  as  the  fraction  of  BIT/ETE  fault  Indications  (or  lack  thereof)  which  are 
erroneous.  FFSI  can  be  represented  by  the  following  model: 


quantity  of  false 

alarms  (QpA) 

quSnTilyof  BIT/) 
Indicated  faults 


quantity  of 
undetected 
faults  (Qud) 

quantity  of 
undetected 
faults  (Qud) 


inspection  of  Tables  11  and  12  indicates  when  false  alarms  and  undetected  faults 
will  occur  (relative  to  the  classification  of  BIT/ETE  faults).  It  will  be  shown 
later  on  (Section  6.2. 1)  that  FFSI  must  be  greater  than  or  equal  to  FFA. 


3.2.5  Mettn  Fuult  Detection  Time  (Tpn) 

The  mean  fault  detection  time  (TpD)  is  the  average  time  It  takes  for  the  BIT/ 
ETE  function  to  detect  and  indicate  a  fault  from  the  time  that  the  fault  has 
occurred.  Tpr)  can  be  represented  by  the  following  model: 

(time  to  detect  and  indicate  the  rn  BIT/ETE  detectable  fault) 


3. 2. 6  Mean  BIT/ETE  Running  Time  (TQ) 

The  moan  BIT7ETE  running  time  (Tg)  la  defined  ns  the  avornge  active  time 
to  perform  a  BIT/ETE  test  routine.  This  can  be  the  average  for  one  toBt,  a 
group  of  tests,  or  all  tests.  T^  can  be  represented  by  the  following  model: 
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>,  (active  running  time  of  the  r  BIT/ETE  teat  routine,  T„.) 

T  ,  'jQ _ 

aB  the  number  of  B1T/ETE  test  routinoe  (Njj) 

The  aetunl  measurement  of  TQ  is  dependent  upon  how  BIT  is  set  up  and  how  the 
user  wants  to  define  T^  Somo  factors  that  must  be  considered  when  determine 
tng  TgRret 

"•  which  BIT/ETE  function(o)  is  T})  being  computed  for? 

•  does  Tg  refer  to  fault  detection,  fault  isolation,  or  both? 

•  are  BIT/ETE  tests  serial  or  overlapping? 

•  are  BIT/ETE  testa  continuous  or  time  shared? 

•  is  TB  the  running  time  with  or  without  a  fault  found  ? 

•  for  fault  detection,  does  the  BIT/ETE  tost  stop  upon  detection  or  con¬ 
tinue  to  the  end  ? 

e  iB  the  BIT/ETE  tost  periodic  or  performed  ns  requested? 


Generally  (and  for  the  purpose  of  this  report)  Tjj  will  be  limited  to  the  following 
oases 

•  fault  detection  only 

e  periodic  BIT/ETE  tests  only 

e  continuous  or  time  shared  BIT/ETE  tosts 

•  serial  BIT/ETE  tests 

•  all  BIT/ETE  functions 

•  with  or  without  fault  found 

•  stop  upon  fault  detection  or  continue  to  end 
8.2.7  Frequency  of  BIT/ETE  Executions  (Fp) 

The  frequency  of  BIT/ETE  executions  is  defined  as  the  frequency  (or  cycling 
rate)  at  which  periodic  BIT/ETE  tests  arc  executed.  This  does  not  apply  to 
BIT/ETE  tests  that  are  executed  only  upon  request.  Fjj  can  be  represented  by 

[the  time  it  takes  to  execute  the  complete  set  of  HIT/ETB 
teBt  routines] 

+  (the  idle  time  botweon  the  execution  of  the  complete  set 
of  BIT/ETK  test  routines] 


A  cycle  is  defined  us  the  time  from  the  start  of  a  given  BIT/ETK  test  until 
the  same  test  is  started  again.  For  eases  where  all  tests  do  not  huve  the  same 
periodicity,  the  oyolo  time  ia  considered  to  bo  the  larger  cycle  time  of  all  the 
BIT/ETE  tests.  For  example.  Figure  4  shows  the  cycle  time  for  two  BIT/ETK 
test  routines.  The  cycle  time  for  the  testa  combined  (Tests  1  ft  2)  is  equal  to 
the  largest  oyole  time  of  the  two  tests,  namely  Tost  #2. 

3.2.8  Test  Thoroughness  (TT) 

Test  thoroughness  is  defined  aa  the  fraction  of  the  equipment/ system  tested 
by  BIT/ETE  relative  to  the  entire  equipment/system,  TT  can  bo  represented  by 

(amount  of  system/oquipmont  tested  by  BIT/ETE) 

TT  — - — - - - - -  . . ^ . . —  .M  . . 

(amount  of  system/equipment  (amount  of  system  equipment  not 
tested  by  BIT/ETE)  tested  by  BIT/ETE) 

There  are  several  different  measures  that  can  bo  used  to  quantify  the  amount 
of  a  system/equipment  tested  or  untested.'  Some  possible  parameters  of  measure 
aroi 

•  failure  rate  (X)  tested  and  untested 

•  number  of  functions  tested  and  untested 

s  number  of  components  tested  and  untested 

•  number  of  faults  tested  and  untested 

It  should  be  noted  that,  if  the  parameter  of  measure  is  the  same  as  is  used  for 
FFD,  the  two  (FFD  ft  TT)  are  not  necessarily  equal.  However,  if  the  two  FOMs 
are  evaluated  at  the  same  level  (e.g. ,  component  failure  mode  level)  then  FFD 
and  TT  will  be  equal. 

3.2.9  Fault  Isolation  Resolution  (FIR(L)) 

Fault  isolation  resolution  (FIK(L))  is  defined  as  the  fraction  of  detectable 
faults  that  can  be  isolated  by  BIT/ETE  down  to  an  acceptable  (specified)  minimum 
number  of  replaceable  Items  (Rls).  FIR(L)  can  be  represented  by 

j  quantity  of  detected  faults  isolated) 

|  to  s  L  Rls  with  BIT/ETE,  (Q1L) 

quantity  of  detected  faults,  (Q^) 

Whon  F1R(L)  is  specified,  it  is  usually  specified  for  more  than  one  L. 
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CVCU'  timi;  or  bit/etk  teit  no.  a 


BIT/ETE  TESTS  NO.  1  4  NO.  1  COMBINED 


o 

BIT/ETC  TEST  NO.  I 


l  MIN  *  Min 

Vi  ¥  / 

CVCU:  TIM*  Or  ilT/ETF.  TRET*  NO.  »  *  NO.  8 
•  CVCI.e  TIME  or  BIT/ETti  TBIT  NO.  I 


I 


Figure  4.  Cycle  Time  fur  Individual  B1T/KTE  Ten  Routine*  ami  Their  Combined 
Cycle  Timea 


8.2.10  motion  of  Fault*  Isolated  by  BIT/ETE  (FFI) 

The  fraction  of  fault*  Isolated  (FF!)  by  BIT/ETE  is  a  generalisation  of 
FIR(L).  FFI  oan  be  defined  as  the  fraction  of  faults,  detected  by  BIT/ETE. 
isolated  b(y  BIT/ETE  to  the  replacement  level  specified  by  the  maintenance  con¬ 
cept.  FFI  oan  be  represented  byt 

quantity  of  detected  faults  isolated  with  BIT/ETE  (Qu 
ppj  .  quantity  of  faults  detected  (Qpr)) 
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3. 2, 11  Mean  Fault  Isolation  Time  (Tpi) 

The  moan  fault  isolation  time  (Tyj)  la  definod  as  the  average  time  to  com¬ 
plete  the  fault  Isolation  process  using  BIT/BITE.  T^j  con  be  represented  by 


lFI 


WBDF 

JT  (time  to  Isolate  the  Ith  fault  with  BIT/ETE) 
quantity  of  faults  detected, 


3.2.12  Maintenance  Personnel  Skill  Level  (MPSL) 

Maintenance  personnel  skill  level  (MPSL)  can  be  defined  too  different  ways. 
One  way  la  to  define  MPSL  as  the  average  skill  level  required  to  perform  cor¬ 
rective  maintenance  for  a  system/equlpmcut,  MPSL  can  also  be  defined  as  the 
minimum  skill  level  (l.e.*  skill  level  with  the  lowest  ability)  required  to  per¬ 
form  corrective  mnintonnnoe  on  n  system  /equipment .  For  the  purpose  of  this 
study,  the  latter  definition  will  ho  used  since  the  nppcnrnnoo  of  this  FOM  in 
specifications  typically  means  that  the  Bkili  level  available  will  be  limited, 


3.2. 13  BIT/ETE  Reliability  <MTBFB  ,K) 

The  BIT/ETE  Hoi  lability  (MTllF  )  ts  defined  ns  the  probability  that  the 
BIT/ETE  circuitry  will  perform  its  intended  function  for  n  specified  Interval 
under  specified  conditions.  BIT/ETE  circuitry  is  any  hardware  that  in  used  for 
BIT/ETE  testing  that  is  not  common  to  the  system  hardware,  MTUF^,  can 
be  represented  byt 


MTBF 


B/K 


V » 

k*l  11/1  k 


wherej  \ 


B/E 


N 


B/E 


failure  rate  of  the  BIT/ETE  hardware  component 

quantity  of  BIT/ETE  hardware  components  not  common 
to  system  hardware 
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3.2.14  BIT/ETE  MftlntainabiUty  (MTTRB/E) 

BIT/ETE  maintainability  (MTTRJE)  la  defined  as  the  average  time  to 
repair  a  fault  In  the  BIT/ETE  circuitry.  MTTftB/E  can  be  expressed  several 
ways.  One  way  of  expressing  MTTRjj^  Is  byt 


MTTRb/e 


N 


B/E 


-JEL 


N 


E 


k-1 


XB/E  R  MCTk 
*B/Ek 


where: 


1b  the  repair  time  for  the  k*h  BIT/ETE  hardware 
component 

*u 

failure  rate  of  the  k  BIT/ETE  hardware  component 


The  above  method  was  extracted  from  RADC-TR-78-109,  Maintainability  Pre¬ 
diction  and  Analysis  Study.  Other  methods  tor  determining  MTTRB  can  be 
found  in  MlL-HDBK-472  and  RADC-TR-70-89. 


3.2.15  BIT/ETE  Availability  AB/E) 

BIT/ETE  availability  (A p/g)  is  defined  ns  a  measure  of  the  degree  to 
whioh  the  BIT/ETE  circuitry  is  in  the  operable  and  committablo  state  at  the 
start  of  u  mission,  when  the  mission  is  called  for  at  an  unknown  (random) 
point  in  time.  AQyfl  onn  be  represented  by: 

mtbpb/b 

ab/e  "  KmiF““+iffrHyE 
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3,2.16  System  Maintainability  (MTTR) 

System  maintainability  (MTTR)  is  defined  as  the  average  corrective  mainte¬ 
nance  time  for  all  systom/equipment  faults.  Ab  with  MTTRg^E,  MTTR  can  bB 
represented  several  ways.  One  way  1st 


MTTR 


£  *1  ^CT 
i»l  1  CTi 

l“l  1 


where:  N  is  the  number  of  components  in  the  system /equpment 
\  i  *  failure  rate  of  the  1*^  component 

Mnm  f  repair  time  for  the  component 
CTi 

3,2,1V  System  Availability  (A) 

System  Availability  (A)  is  defined  ns  a  measure  of  the  degree  to  whioh 
a  system  (or  equipment)  Is  In  the  operablo  and  committable  state  at  the  start 
of  a  mission,  whon  the  mission  is  called  for  at  an  unknown  (random)  point  in 
time.  "A"  can  bo  oxprossod  tho  same  way  as  AD^. 

MTBF 

A  »  MTW-f  MTTR 


3. 3  CATEGORIZATION  OF  THE  IDENTIFIED  BIT/ETE  FOMs 

The  BIT/ETE  FOMa  defined  in  Seotion  3, 2  were  categorized  according  to 
the  BIT/ETE  objectlve(s)  (identified  in  Section  3. 1)  that  they  fulfill. 

Categorization  of  the  FOMa  defines  how  each  identified  FOM  characterizes 
BIT/ETE  and  aids  in  determining  what  new  FOMa  may  be  needed.  Categorization 
of  the  FOMa  was  also  used  in  Seotion  8. 0  for  determining  whloh  FOMa  are  inter** 
related  and  for  determining  an  appropriate  set  of  FOMs  that  should  be  apeoified 
for  a  given  application.  Figure  S  indicates  those  BIT/ETE  FOMs  defined  in  Seo¬ 
tion  3.  U  that  are  associated  with  each  BIT/ETE  objective.  Some  BIT/ETE  FOMs 
appear  under  more  than  one  BIT/ETE  objective  since  they  can  characterize  more 
than  one  facet  of  BIT/ETE.  Also,  it  should  be  noted  that  not  all  the  FOMs  asso¬ 
ciated  with  any  one  BIT/ETE  objective  can  be  used  to  precisely  quantify  the  ob¬ 
jective  or  requirement.  These  particular  FOMs  are  either  qualitative  measures 
or  indirect  quantitative  measures  of  the  same  objective.  However,  they  can  still 
be  used  to  impose  requirements  on  the  associated  BIT/ETE  objectives. 
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3.4  IDENTIFICATION  OF  NEvf  IttT/ETE  FOMs  REQUIRED 

The  purpose  of  this  task  was  to  Identify  any  new  BIT/ETE  FOMs  required  to 
fill  in  the  voids  in  Figure  5  where  a  given  BIT/ETE  objective  is  not  covered  by 
an  existing  FOM,  Inspection  of  Figure  5  indicates  that  all  the  BIT/ETE  objectives 
have  an  associated  FOM(s) .  Whether  these  FOMs  are  good  measures  of  the  ob¬ 
jective  is  unknown  (this  will  be  determined  later  on  in  Section  4). 

One  area  that  may  require  a  now  FOM  is  the  BIT/ETE  objective,  Fault 
Isolation  Aoouracy.  The  Identified  BIT/ETE  FOM  that  quantifies  this  objective 
the  best  is  FIR(L).  However,  FIR(L)  is  more  of  an  indication  on  how  thorough 

i 

the  BIT/ETE  fault  Isolation  function  is  rather  than  accuracy. 

A  possible  new  FOM  may  be  the  False  Pull  Rate  (FPR)  or  the  Fraction  of 
False  Pulls  (FFP).  FPR  (or  FFP)  is  defined  as* 

FPR-  The  rate  at  which  good  RIs  (1.  e. ,  RIs  with  no  actual  failure  within  it) 
are  removed  from  a  system  due  to  the  result  of  a  BIT/ETE  fault 
isolation  process, 
or 

FFP  -  The  fraction  of  RIs  removod  from  a  system,  duo  to  the  result  of  a 
BIT/ETE  fault  isolation  process,  that  are  good  RIs  (l.e. ,  RIs  with 
no  actual  failure  within  It). 

quantity  of  good  RIs  removed  (Qnm) 

ppp  n  _  -  . . .  i 

quantity  of  RIs  romovea  (Qj^) 

The  only  problem  associated  with  the  use  of  FPR  or  FFP  is  that  it  is  depend¬ 
ent  upon  the  maintenance  concept.  That  is,  a  system  with  a  certain  amount  of 
BIT/ETE  may  have  two  different  values  for  FPR  (or  FFP)  given  two  diffei*ont 
maintenance  concepts.  For  example,  assume  a  system  with  an  average  fault 
isolation  resolution  (i.e. ,  average  BIT/ETE  fault  Isolation  group  sUe,  1)  of 
three  RIs.  If  the  system's  maintenance  concept  is  "RI  group  replacenent,"  then 
tho  FPR  would  bo  two  out  of  every  three  RIs  or  0.07  for  FFP  (i.e, ,  on  the  aver¬ 
age  two  good  RIs  are  removod  for  ever)’  bad  one).  On  the  other  hand,  if  the 
maintenance  concept  is  "Iterative  RI  replacement"  (l.e.,  removo/replace  RIs 
one  at  a  time  until  the  fault  is  corrected) ,  then  tho  FPR  would  be  one  out  ct  two 
on  the  average  (l.e, ,  average  number  of  iterations  required  to  correct  the  fault 
is  two)  or  0. 30  for  FFP.  It  should  bo  noted  here  that  FIR(L)  and  FFP  arc  very 


similar.  As  a  matter  of  fact,  FFP  can  be  derived  exactly  through  FIR(L>.  For 
example,  for  any  given  set  of  FIR(Lj)  <e.g, ,  0.90  s  1  RI,  0.95  as  3  RIs,  and 
1.0  a  10  RIs),  an  average  fault  tsolatlon  group  size  oan  be  determined,  8  (i.e. , 
S  is  the  average  RI  group  slse  that  a  fault  oan  be  isolated  to).  Depending  upon 
the  maintenance  concept,  FFP  oan  easily  be  determined  byt 

l  •' 

if  tlie  maintenance  concept  was  RI  group  replacement, 

FFP  *  1*JL 


if  the  maintenance  concept  was  Iterative  RI  replacement, 


ff-1 


rrr 


Thus,  for  the  remainder  of  this  report,  FPR  &  FFP  will  be  excluded  since 
they  oan  be  derived  from  FIR  (L). 

One  other  possible  now  BIT/ETE  FOM  Is -the  erroneous  fault  Isolation  rate 
(EFIR),  or  the  fraction  of  erroneous  fault  isolation  results  (FEFI).  EFIR  and 
FEFI  are  defined  asi 

EFIR  -  The  rate  at  whioh  a  BIT/ETE  fault  isolation  process  results  in 

identifying  the  wrong  RI  (i.  e. ,  fault  isolation  prooess  results  In  a 
suspe' t  group  of  RIs,  but  when  the  RIs  are  replaced,  the  fault  still 
exists)  once  a  fault  has  been  detected. 


FEFI  -  The  fraotion  of  BIT/ETE  fault  isolation  results  that  Identify  the 
wrong  RI  onoe  a  fault  haB  been  detected. 

quantity  of  erroneous  fault  isolation  results,  (QEFIR) 

FEFI  "  quantity  of  fault  isolation  results, 

EFIR  and  FEFI  may  be  good  ohoioes  of  BIT/ETE  FOMs  to  measure  the  ap- 
curacy  of  BIT/ETE  fault  Isolation  since  they  essentially  measure  how  well  the 
BIT/ETE  fault  isolation  function  has  been  documented  (i.  e. ,  the  number  or  rate 
of  erroneous  fault  isolation  results  is  largely  dependent  upon  how  well  the  BIT/ 
ETE  is  documented  in  maintenance  manuals  and  the  software).  Note  that  the 
software  or  the  maintenance  manuals  used  to  present  fault  Isolation  results  is 
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also  a  part  of  the  BIT/ETE,  since  a  BIT/ETE  function  that  is  capable  of  isolating 
faults  to  a  single  Rt  1h  of  nu  good  if  the  results  cannot  be  presented  to  the  user 
correctly, 

For  the  remainder  of  this  report  only  FEFI  (the  fraotion  of  erroneous  fault 
isolation  results)  will  be  addressed*  since  EFIR  is  similar. 


The  objective  of  this  task  wee  to  evaluate  the  suitability  of  the  FOMs  defined 
In  the  preceding  eeotlon  as  design  specifications. 

The  approach  used  to  evaluate  tho  FOMs  suitability  was  a  weighted  scoring 
technique.  The  steps  Involved  In  the  approach  were* 
l)  establish  an  evaluation  criteria 

8)  score  each  FOM  according  to  the  established  evaluation  criteria 
8)  determine  the  scores  for  each  FOM  by  a  weighted  soorlng  sum. 

4  The  following  subseotions  contain  the  detailed  information  on  the  accomplish 
ment  of  the  above  tasks. 
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4.1  EVALUATION  CRITERIA 

The  faotors  used  to  determine  the  suitability  of  a  BIT/ETE  FOM  for  use  as 
possible  design  specifications  arei  1)  ambiguity,  2)  trnnslatablllty,  3)  traokabll- 
Ity,  4)  demonstratabillty,  5)  applicability,  and  6)  uniqueness,  A  FOM  must  be 
unambiguous  to  establish  a  common  baseline  of  what  the  FOM  represents.  A 
FOM  must  be  translatable  so  the  design  engineer  can  Interpret  the  specifications 
Into  terminology  and  quantitative  requirements  familiar  to  him.  It  must  also  be 
traokable  so  It  oan  be  evaluated  during  the  various  stages  of  a  system’s  develop¬ 
ment  to  determine  how, much  BIT/ETE  has  been  designed  into  a  system  and  how 
effective  it  Is.  A  FOM  must  be  demonstratabie  so  the  BIT/ETE  oan  be  evaluated 
to  see  If  the  requirements  have  been  met.  A  FOM  should  be  applicable  to  all 
system  types  In  order  to  reduoo  the  quantity  of  unique  FOMs  used.  Finally,  a 
FOM  should  characterise  as  many  of  the  BIT/ETE  characteristics  as  possible, 

In  order  to  minimise  the  amount  of  BIT/ETE  FOMs  required  for  a  single 
specification. 

The  six  suitability  faotors  mentioned  in  the  previous  paragraph  wore  used 
as  the  scoring  faotors  In  the  evaluation  process.  The  preolso  definitions  of  each 
suitability  factor  worot 

1)  ambiguity  -  This  scoring  factor  was  used  to  determine  how  definitive 
a  FOM  Is.  That  is,  oan  n  FOM  be  Interpreted  Into  Its  true  definition 
without  any  difficulty,  or  can  the  definition  be  Interpreted  more  than  one 
way.  The  more  definitive  a  FOM  Is,  the  higher  it  scored, 

2)  translatabtllty  -  This  scoring  factor  was  used  to  determine  how  good  a 
FOM  Is  to  the  engineer  as  a  design  tool.  In  other  words,  oan  the 
engineer  take  the  specified  FOM  and  translate  It  Into  parameters  that 
oan  be  used  In  hts  design  effort.  The  more  useful  a  FOM  is  to  the 
designer,  the  higher  It  scored. 

3)  traoknblUty  -  This  scoring  factor  was  used  to  measure  a  FOM's  stability 
to  be  quantified  and  evaluated  during  a  system's  various  phases  (e.  g. , 
conceptual,  design,  development,  , . . The  easier  It  was  to  quantity  a 
FOM  during  these  various  phases,  the  higher  It  scored. 
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4)  demonstratabllity  ~  This  scoring  faotor  measures  a  FOM's  ability  to  bo 
quantitatively  verified.  This  pertains  to  either  formal  demonstration, 
Informal  demonstration,  or  field  usage.  The  easier  it  was  to  quantify 

i  . 

a  FOM  via  testing,  the  higher  it  scored, 

5)  applicability  *-  This  sodring  factor  Was  used  to  detormiho  a  FOM's 
usefulness  towards  Various  system  types.  The  more  system  types  a 
FOM  was  applicable  to,  the  higher  it  soored. 

8)  uniqueness  -  This  scoring  factor  was  used  to  measure  the  overall 
effectiveness  of  specifying  a  FOM.  That  is,  the  more  BIT/ETE  objec¬ 
tives  (fault  detection  and  fault  isolation  were  separated)  a  FOM  char¬ 
acterised,  the  higher  It  scored. 

Onoe  the  scoring  factors  were  determined,  weights  were  assigned  to  them 
according  to  their  importanoe  in  BIT/ETE  FOM  specification.  The  soorlng 
weights  assigned  to  each  factor  are  summarised  In  Table  13  below, 

TABLE  13.  SCORING  FACTOR  WEIGHTS 


Suitability  Faotor 

Weight 

Ambiguity 

.12 

Translatablllty 

17 

Traakablltty 

20 

Demonstratabllity 

18 

Applicability 

8 

Uniqueness 

25 

The  rationale  for  selecting  the  weights  was  as  follows! 

1)  Uniqueness  -  If  a  FOM  did  not  characterise  BIT/ETE  (i.e. ,  the  BIT/ 
ETE  performance  capabilities),  it  was  of  little  use,  thus  uniqueness 
was  oonslderod  the  most  important  faotor.  (The  goal  was  to  find  a 
minimum  set  of  FOMs. ) 

2)  Translatablllty,  tr&ctabillty,  and  demonstratabllity  were  assigned 
relatively  the  same  weight,  A  FOM  must  be  capable  of  being  translated 
Into  design  parameters,  followed  (trnoked)  through  the  various  design/ 
development  stages,  and  verified  through  testing. 

3)  Finally,  ambiguity  and  applicability  were  given  the  smallest  wetghtB 
since  theso  two  faotors  were  not  necessities,  (i.e. ,  it  would  be  nice 
to  have  a  FOM  that  was  not  ambiguous  and  was  applicable  to  all  system 
types,  but  a  FOM  could  still  be  useful  without  these  characteristics. ) 


With  the  evaluation  criteria  mentioned  previously!  It  was  possible  to  evaluate 
the  FOMs  using  the  following  model; 

6 

, ..  ,,  ,  -  ,  evaluate  score  -  £  -  W.S, 

' ' J  ’  (  '  1*1  1  1 

whsrei . 

Wj  *  scoring  weight  of  the  1  factor 

a  ------  il.  ith 


8|  *  score  assigned  to  the  1  factor  ■■  •  . 

i  «  soorlng  factor  (ambiguity,  translatabllity. . . ) 

The  results  of  the  evaluation  process  are  summarised  in  the  next  section. 
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4.2  BIT/ETE  EVALUATION  RESULTS 

Prior  to  evaluating  the  B1T/ETE  FOMs,  they  were  broken  up  Into  the 
mijor  categories  Identified  In  Section  3.  The  categories  in  which  they  were  1 
grouped  were:  1)  BIT/ETE  FOMa  that  characterise  the  BIT/ETE  fault  detec¬ 
tion  capabilities,  2)  BIT/ETE  FOMa  that  characterise  the  BIT/ETE  fault  isola¬ 
tion  capabllltlea ,  and  3)  BIT/ETE  FOMa  that  characterise  the  operational 
charaotoriatloa  of  BIT  /ETE .  Thia  waa  done  to  avoid  the  compariaon  of  FOM 
types  that  were  not  applicable  to  the  same  FOM  characteristic  (l.e. ,  the  unique¬ 
ness  suitability  factor  was  scored  for  either  fault  detection  capability  or  fault 
isolation  capability ,  but  not  both).  FOMa  that  measured  BIT/ETE  operational 
characteristics  were  scored  on  the  baals  of  tho  first  flvo  suitability  faotors 
alone,  since  uniqueness  is  not  applicable  to  those  FOMs  (these  FOMs  wore 
scored  aero  for  this  factor). 

Table  14  ia  a  summary  of  the  ^valuation  rosults.  A  majority  of  tho  BIT/ 
ETE  FOMa  scored  as  expected,  but  there  wero  a  few  surprises.  Fraction  of 
False  Alarms  (FFA)  and  Fraction  of  false  Status  Indications  (FFSI)  wore  ox- 
peoted  to  score  high  but  actually  scored  very  low.  This  was  largely  due  to 
their  Inability  to  be  translated,  tracked  and  demonstrated  with  relative  ease. 
The  new  FOM  identified,  Fraotion  of  Erroneous  Fault  Results  (FEFI)  also 
noored  low  fbr  the  some  reasons. 

The  BIT/ETE  FOMs  that  scared  well  were  RMA  1‘OMs  that  are  currently 
specified  (A  and  MTTR).  Fraotion  of  Faults  Detooted  (FFD)  and  Fault  Iso¬ 
lation  Resolution  (F1R(L))  also  scored  relatively  high  hb  expootod. 

The  results  of  the  BIT/ETE  FOM  evaluation  provide  the  basis  for  Sec¬ 
tion  6,  the  FOM  guidelines  section,  in  determining  which  FOMs  should  be 
specified  over  others.  The  overall  objective  was  to  determine  which  FOMs 
were  the  best  to  specify.  However,  this  can  not  automatically  rule  out  FOMs 
that  soored  low  (t.e. ,  FFA,  FFSI,  . , .)  in  the  evaluation  prooess  since  the 
FOMa  that  are  aotually  selected  ore  also  dependent  upon  the  BIT  / ETE  objec¬ 
tives  that  aro  desired. 
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FOMs 

8 

8*  W 

8 

8*  W 

8 

8  *W 

8 

S*W 

□1 

Fault  Detection  Capability 
1.  FFD 

B 

96 

9 

183 

9 

180 

4 

72 

9 

2. 

?! 

4 

48 

8 

136 

8 

160 

8 

144 

9 

3. 

5 

60 

9 

153 

9 

180 

2 

36 

9 

4. 

3 

36 

7 

119 

8 

180 

8 

144 

9 

s. 

5 

60 

7 

119 

8 

160 

2 

30 

9 

6. 

FFSI 

5 

60 

4 

68 

3 

80 

8 

90 

9 

7. 

FFA 

5 

60 

3 

51 

1 

20 

3 

54 

9 

Fault  Isolation  Capability 

1.  MTTR 

B 

96 

8 

136 

8 

160 

9 

162 

9 

2. 

A 

7 

84 

8 

136 

8 

160 

9 

182 

9 

3. 

FIR(L) 

9 

108 

8 

88 

7 

140 

6 

108 

9 

4. 

FFI 

7 

84 

6 

85 

7 

140 

6 

108 

9 

8. 

TT 

B 

60 

9 

183 

9 

180 

2 

36 

9 

6. 

8 

60 

6 

102 

8 

100 

8 

144 

9 

7. 

6 

72 

3 

81 

2 

40 

5 

90 

9 

8. 

MPSL 

1 

12 

1 

17 

1 

20 

3 

54 

9 

BIT/ETE  Characteristic** 

i.  mtbfb/e 

7 

84 

9 

133 

9 

180 

9 

162 

9 

2. 

mttrb/e 

8 

96 

8 

136 

8 

160 

9 

162 

9 

3, 

Ab/E 

7 

84 

8 

130 

8 

160 

9 

162 

9 

l-W 

8 

8  »W 

2(8.  W)' 

72 

6 

180 

723 

72 

4 

100 

660 

72 

5 

125 

626 

72 

3 

75 

606 

72 

5 

125 

572 

72 

6 

150 

800 

72 

2 

80 

307 

72 

0 

150 

776 

72 

4 

100 

714 

72 

7 

175 

688 

72 

6 

180 

639 

72 

3 

125 

626 

72 

5 

125 

003 

72 

4 

100 

425 

72 

1 

28 

200 

72 

0 

051 

72 

0 

M 

626 

72 

0 

614 

Ab  stated  previously  in  the  Evaluation  Section  (Section  4),  two  of  the  moat 
important  aspects  of  BIT/ BTE  FOMa,  are  a  FOM'a  ability  to  be  analysed 
and  demonstrated.  Thii  section  presents  the  models  and  techniques  that  have 
beeti  developed  (or  existing  techniques)  to  analyse  and  demonstrate  BIT/ETE  FOMs. 
The  remainder  of  this  seotion  dtaoueeee  the  techniques  developed  and  how  to  im¬ 
plement  them. 


li.l  BIT/KTE  POM  ANALYSIS  TECHNIQUES 

It  is  evident  that  In  order  to  have  a  high  confidence  in  the  amount  of  BIT/KTK 
that  has  been  designed  into  a  system,  a  rigorous  and  detailed  analysis  must  be  per¬ 
formed.  It  has  been  determined  that  a  majority  of  the  BIT/ETE  POMs  identified 
in  Seotton  3  can  be  analysed  using  either  existing  ItMA  FOM  analysis  meth¬ 
odologies  or  similar  techniques.  The  analysis  techniques  developed  (or  existing) 
can  be  divided  into  three  distinct  groups) 

1)  rate  (e.g.  failure  rate)  dependent  techniques 

2)  time  dependent  techniques 

3)  rate  and  time  dependent  techniques. 

Table  13  summarizes  the  POMs  associated  with  each  technique.  The  rate  de¬ 
pendent  technique  applies  to  FOMs  whose  numerics  can  be  determined  solely  by 
the  ratio  of  rate  of  occurrences  of  aomo  ovent(s)  (e.g.,  fa  Hurt'  rate  tented  divided 
by  the  total  failure  rate  of  a  system  can  be  used  to  express  test  thoroughness), 

POMs  that  pertain  to  time  dependent  analysis  techniques  aro  POMs  that  are  strictly 
measured  or  determined  through  time  synthesis  (t.e.,  standard  elemental  mainte¬ 
nance  time  tables).  Finally,  the  POMs  that  are  determined  by  a  combination  of 
the  two  previous  techniques  are  FOMs  that  are  moasurod  through  time  synthesis, 
but  are  averaged  by  a  rate  of  occurrence  weighting, 

Availability  (A  and  AB/K),  BIT/ETE  Reliability  (MTBFB/E),  and  Maintenance 
Personnel  Skill  Level  (MPSL)  arc  not  included  in  Table  15.  Availability  and  BIT/ETE 
Reliability  can  be  analyzed  using  olassieal  teehniquea  and  their  definitions  presontod 
in  Sootlon  3,2.  Due  to  the  uniqueness  of  MPSL,  it  is  discussed  separately  in  Section 
li.l. 4.  The  analysis  techniques  applicable  to  each  FOM  are  discussed  in  the  follow¬ 
ing  subsections. 

TABLE  15.  ANALYSIS  TECHNIQUES  AND  THE  BIT/ETE  FOMs 
APPLICABLE  TO  THEM 


Rate  Dependent 


Time  Dependent 
FOMs 


Rate/Time  Dependant 


rr 


5,1.1  Rate  Dependent  Analysts  Techniques 

Prom  the  POM  definitions  of  Section  3.2,  it  is  evident  that  Fractional  Faults 
Detected  (FFD),  Fraction  of  False  Alarms  (FFA),  Fraction  of  False  Status  Indications 
(FFSI),  Teat  Thoroughness  (TT),  Fault  Isolation  Resolution  (F1R(L)),  Fraction  of 
Faults  Isolated  (FF I),  and  Fraction  of  Erroneous  Fault  Isolation  Results  (FEFI)  can 
be  expressed  as  ratios.  Typically  the  events  being  described  are  failures  and  the 
rate  of  occurrence  is  expressed  as  the  failure  rate.  In  the  case  of  FFA  and  FFSI 
the  occurrence  rate  includes  items  other  than  failures)  however,  for  simplicity  the 
following  discussion  will  address  only  failure  rate.  Details  on  how  the  other  rates 
(e.g.  false  alarm)  are  included  in  the  analysis  are  described  in  the  appropriate 
subsections. 

Formulation  of  each  respective  ratio  is  accomplished  by  determining  the  fail¬ 
ure  rate  associated  with  the  numerator  of  the  respective  FOM  definition  (refer  to 
Section  3.2)  and  dividing  it  by  the  failure  rate  associated  with  the  denominator  of 
the  FOM. 

The  general  form  for  analyxiiqt  BIT/ETE  FOMs  expressed  as  ratios  1st 


ratio  = 


Nc 

v  \ 
|.l  x'< 


Nc 

i  «'  1 


>  ~  Xd, 


Whore) 

xnt  -  failure  rate  of  the  ^component  associated  with  the  numerator  of 
the  defined  FOM 

\ij  -  failure  rato  of  the  l**'  component  associated  with  the  denominator 
of  the  defined  FOM 

N0  -  number  of  components  in  the  system 

The  failure  rate  associated  with  the  numerator  and  denominator  of  each  ratio 
can  bo  determined  by  summing  up  the  associated  failure  rates.  Summation  o*  the 
falluro  rates  is  dependent  upon  the  level  of  analysts.  FOM  analyses  can  be  per¬ 
formed  at  any  desired  indenture  levol  (o.g.,  component  failure  mode  level,  com¬ 
ponent  level,  functional  lovel,  unit  level),  it  is  clear  that  the  most  accurate  results 
will  be  obtained  for  the  analysis  performed  with  the  most  dotall  (l.e.  component 
failure  mode  level),  For  simplicity,  the  model  presented  is  for  the  oomponont 
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level  analysis.  The  models  for  the  other  levels  oan  easily  be  obtained  by  slight  modi¬ 
fications.  Table  18  summarises  the  various  forms  of  the  models  that  can  be  used 
for  analyzing  FOMs  expressed  as  ratios.  For  the  remainder  of  this  report,  only 
the  models  fofr  the  component  level  analysis  will  be  presented. 

The  moat  dlffiouit  part  of  analysing  FOMs  expressed  as  ratios  is  the  deter¬ 
mination  of  the  corresponding  numerator  and  denominator  failure  rates.  AN  FMEA 
(failure  modes  and  effects  analysis),  along  with  the  BTT/ETE  teat  philosophy,  oan 
be  used  to  facilitate  the  Identification  of  these  failure  rates.  The  depth  of  the  FMEA 
in  dependent  upon  the  analysis  level  selected.  The  following  subsections  discuss 
the  application  of  the  models  developed  to  express  FOMs  as  a  ratio  of  failure  rates 
for  the  associated  FOMs. 

5.1. 1.1  Method  of  Analysis  for  Fraotion  of  Faults  Detected  (FFD) 

Aa  stated  in  Seotlqn  3.2.2,  FFD  can  be  expressed  two  different  ways}  therefore, 
two  models  are  presented  here  for  the  analysis  of  FFD.  The  models  developed  for 
FFD  aret 

e  For  FFDa,  (all  faults)) 


SI 


TABLE  16,  SUMMARY  OF  THE  OCCURRENCE  RATE  RATIO  MODEL  FORMS* 


Analysis  Level 

Model 

Parameters 

Unit  (or  HI) 
Level 

nh 

X  \n 

ratio  a  i-2JL - X 

nr 

X  \A 

J*1  “j 

Nj|  -  number  of  RIs  in  the  ay  stem 

XH  -  Failure  rate  of  the  jth  unit  (or  Rl)  assooi- 
Qj  ated  with  the  denominator  of  the  FOM 

X  -  Failure  rate  of  the  unit  (or  RI)  associ- 
nj  ated  with  the  numerator  of  the  FOM 

•Normally  (X^  a  Xn^) 

Functional 

Lovel 

nf 

ratio  *  - — 1 - 

nf 

k£-l  H 

N.,=  number  of  functions  contained  in  the 
*  syatem 

iL 

X*  -  Failure  rate  of  the  k™  function  associ- 
”k  ated  with  the  denominator  of  the  FOM 

x  -  Failure  rate  of  the  k**'  function  assoct- 
llk  ated  with  the  numerator  of  tho  FOM 

•Normally  (X^a  \n^) 

Component 

Level 

N°  v 

X  \ 

ratio  *  - 

Nc 

2  xd» 

1  =  1* 

N  -  number  of  component*  contained  in  tho 
system 

X  d  -  Failuro  rate  of  the  ith  component  assooi- 
t  ated  with  the  denominator  of  the  FOM 

Xn  -  Failure  rate  of  tho  ith  component  associ- 
i  ated  with  the  numerator  of  the  FOM 

•Normally  (X ^  a  Xn  ) 

Component 

Failure 

Mode 

Lovol 

Nc  Nl 

X  2  XU|„ 
ratio  =i±^1  J? 
N0  Nt 

S  2xdln 

i»l  q»t 

N|  -  number  of  failuro  modes  ns  anointed  with 
the  l*'1  oomponent 

X  -  Failure  mode  of  tho  qtl;  failuro  mode 
t,1iq  of  tho  l*1'  component  associated  with 
tho  denominator  of  tho  TOM, 

-Failure  rate  of  tho  q^v  failure  mode 
of  the  i”1  component  naaooiatod  with 
tho  numerator  of  tho  FOM. 

NOTEt  For  simplicity  the  rates  are  expressed  as  failure  rate. 
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where! 


xunl" 


1L 

The  failure  rate  of  the  i  component 

Tho  failure  rate  of  the  component  that  la  detected  by  BIT/ETE 
X  ^  i  X  j  for  all  i 

llu'  failure  rate  of  the  i^  component  that  can  bo  detected  by  any  mennB 
(*D[  +  *ud  )  *  X  i  ior  ttl1 4 


N0  -  The  number  of  components  in  the  system/equipment. 

Analysis  of  FFD  requires  a  thorough  understanding  (down  to  the  level  of  the 
analysis)  of  how  the  BIT/ETE  will  function  and  what  the  BIT/ETE  can  test.  In 
conjunction  with  an  FMEA,  it  can  be  determined  whether  the  BIT/ETE  can  test 
and  detect  the  various  failure  modes  that  will  occur.  The  failure  rate  of  any  fail¬ 
ure  mode  that  can  be  tested  and  detected  by  BIT/ETE  will  oomprlao  the  numerator 
portion  of  the  FFD  failure  rate  ratio. 

As  mentioned  previously,  tho  level  at  which  tho  analysis  is  performed  will 
affect  the  accuracy  of  the  analysis  results.  Undetected  failures  primarily  oocur 
at  the  component  failure  mode  level,  thus  in  order  to  analyze  them  accurately  the 
analysis  level  should  be  at  the  failure  mode  level.  For  FFD,  it  is  recommended  that 
the  analysis  be  performed  at  the  component  level  or  the  component  fatluro  mode 
level  in  order  to  obtain  aoourate  results. 

5. 1.1. 2  Method  of  Analysis  for  Fraotlon  of  False  Alarms  (FFA) 

AnnlyBlB  of  FFA  for  a  system  is  a  very  difficult  task  (ns  was  indicated  by  Its 


evaluation  score  of  Section  4).  Au  noted  earlier ,  threo  of  the  major  causes  of 


false  alarms  arm  1)  BIT/ETE  fniluroa,  2)  transients  (noise),  and  3)  out-of- 
toloronoo  conditions.  The  difficulties  that  exist  are  in  the  analysis  of  false  alarm 
conditions  caused  by  transients  (noise)  and  out-of-tolerance  conditions. 


In  order  to  analyze  transients,  an  analysis  not  common  with  current  RMA 
analysis  techniques  must  be  performed.  Analysis  of  transients  consists  of  analyz¬ 
ing  the  system's  performance  (analytically)  when  transients  arc  simulated  at  various 
points  in  the  system  and  at  the  systems  interfaces.  This  can  be  done  with  relative 
ease  by  the  use  of  computer  aided  circuit  analysis  techniques.  However,  the 
problem  arises  In  defining  tho  scenario  of  transients  to  be  analyzed  (i.c.,  what  are 
the  expected  transient  conditions  in  terms  of  frequency,  magnitude,  duration, 
etc. ..)» 
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Analysis  of  out-of-tolerance  condition  r quires  an  analysis  similar  to  a  worst 
ease  analysis.  Computer  aided  circuit  analysis  techniques  usually  provide  for  worst 
case  analysis.  However,  these  programs  usually  analyze  an  output  signal  when 
all  the  components  are  at  their  maximum  or  minimum  specified  value.  In  order 
to  perform  out-of-tolerance  analyses  of  this  type,  slight  modifications  must  be 
made  to  the  current  computer  aided  circuit  analysis  programs  available.  These 
modifications  consist  of  providing  the  computer  program  the  capability  to  analyze 
circuits  when  a  single  component  is  out-of-tol trance  or  any  combination  of  com¬ 
ponents  taken  together  are  out-of-tolerance.  It  should  be  noted  here  that  out-of¬ 
tolerance  conditions  that  result  in  false  alarms  are  usually  a  sign  of  poor  design. 

If  out-of-tolorance  conditions  are  analyzed  using  the  method  presented,  then  poor 
tolerance  specifications  will  probably  be  weeded  out  before  the  system  is  fabri¬ 
cated.  Thus,  the  use  of  out-of-tolerance  analysis  techniques  will  provide  feedbaok 
to  correct  problems  before  they  can  occur.  In  the  event  that  a  computerized  cir¬ 
cuit  analysis  program  is  not  available,  the  above  analysis  would  be  next  to  impos¬ 
sible  exoept  for  a  very  simple  equipme’ t . 

The  previously  mentioned  analyses  (transient  and  out-of-tolerance)  are  fine 
for  determining  which  components  promote  false  alarms,  but  the  methods  presented 
cannot  aid  the  engineer  in  determining  when  or  at  what  rate  these  false  alarm 
conditions  will  occur.  In  order  to  do  this,  methods  must  bo  derived  to  determine 
the  drift  characteristics  (and  probabilities  of)  and  the  noise  immunity  character¬ 
istics  (and  probabilities  of)  of  system  components  ar  well  as  transients  oaused  by 
external  sources,  For  this  study  no  attempt  has  been  made  to  determine  these 
methods.  For  the  time  being,  engineering  judgment  must  be  used  until  methods 
can  be  developed. 

The  analysis  of  FFA  is  dependent  upon  the  system  reliability  philosophy 
(i.o. ,  are  B1T/ETE  failures  also  considered  system  failures).  If  BIT/ETE  failures 
are  considered  system  failures,  then  FFA  can  be  analyzed  byt 


Nc  x 

£  xd,  +  «  +  y 

i  *  1 
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whoret  th 

Xnt  “  The  BIT/ETE  detectable  failure  rate  of  the  i  component 

(see  5. 1.1.1) 

6  -  the  rate  at  which  out-of-tolerance  conditions  will  occur 

y  -  the  rate  at  which  transients  occur 

Nq  -  number  of  components  in  the  system 

If  BIT/ETE  failures  are  independent  of  system  failures,  then  FFA  can  be 
analyzed  by  the  following  model) 

FFA9  a  L 11 - — 

Nrt  Nn~ 

E  \  +  X?  V  +  d  +  v 

i=l  1  j=l  J 

where) 

NBC  "  t'10  number  of  BIT/ETE  dedicated  components 

-  failure  rate  of  the  jth  BIT/ETE  component  that  results 
in  a  false  alarm 

NOTE)  For  fie  two  models  presented,  the  number  of  system  components 
Nq,  is  not  equal  slnoe  NQ  for  the  second  model  does  not  include 
BIT/ETE  dedicated  components. 

(l.o.,  Nq  for  model  l  is  equal  to  Ne  +  nBC  of  model  2) 

5. 1, 1.3  Method  of  Analysis  for  Fraction  of  False  Status  Indications  (FFSI) 

Since  FFSI  also  includes  FFA,  the  methods  used  to  determine  false  alarms 
(Section  5. 1.1. 2)  also  apply  when  determining  FFSI.  The  model  for  analyzing  FFSI 

iS! 


FFSI  = 

E  (xi)j +  xud1)  +  a  +  y 

ii 

til 

is  the  failure  rate  of  the  I  component  that  cannot  be 
detected  by  any  means. 

<  *D(  +  ^  a  \  for  aU  1 
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wherei 


NOTE) 


£  +  <S  +  Y 


l 

I 
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•rJfci.  *> 


Note  that  the  model  presented  1b  for  the  01100  where  B1T/ETK  failures  are 
conBldered  Bystem  failures  (analogous  to  FFAi  In  5.1, 1,2).  If  BIT/i)TF  failures 
are  independent  system  failures,  then  the  summations  of  X^p  are  included  in  the 
numerator  and  denominator,  and  the  summations  of  nnd  ^UDj  are  modified  to 
exclude  BIT/KTE  components. 

5.1. 1,4  Method  of  Analysis  for  Test  Thoroughness  (TT) 

As  noted  before,  there  are  several  ways  to  analyze  the  FOM  TT.  Using  a 
ratio  of  failure  rates  is  the  most  effective  measure.  Tho  model  for  analyzing  TT 
1st 


TT  o 


N 


Vp 


No 


wheroi 

Ny  -  number  of  components  In  the  system 
^  -  failure  rate  of  the  1th  component 

-  failure  rate  of  the  i1*'  component  that  Is  "tested"  by  MT/Ftk, 

1  (\Tj[  Xi  tlU'  1) 

The  actual  usefulness  of  the  TT  FOM  Is  largely  dependent  upon  the  level  of 
analysis  (ns  was  true  with  FFD).  The  above  model  presents  TT  for  the  component 
analysis  level,  one  of  the  more  detailed  analysis  levels.  The  analysis  of  TT  can  be 
performed  at  higher  levels,  such  as  the  functional  level  and  the  equipment  level. 
However,  the  accuracy  of  the  FOM  goes  down  as  the  analysis  level  gets  higher. 
This  is  duo  to  the  assumptions  made  when  a  function  is  considered  tested.  That  is, 
when  a  function  is  classified  as  "tested"  by  BIT/BTE,  the  assumption  made  is  that 
the  entire  function  (i.e.,  all  components,  thus  tho  entire  failure  rate)  is  tested. 
Using  these  assumptions  will  give  inaccuracies  in  the  measure  of  TT. 

The  numerator  of  the  model  for  TT  consists  of  the  failure  rate  "tested"  by 
B1T/ETE.  There  is  a  great  deal  of  uncertainty  as  to  what  "tested"  means.  The 
definition  of  tested  is  dependent  upon  the  level  that  the  analysis  is  performed  on. 
Tho  most  general  definition  for  tested  ist 


"Tested"  *  The  portion  of  the  equipment  that  is  monitorod  by  BIT/ETE. 
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A* , 


If  the  analysis  is  performed  on  the  component  failure  mode  level,  then  it  is 
safe  to  say  that  the  failure  rate  of  the  components  tested  (or  monitored)  is  also 
the  failure  rate  of  the  faults  deteotable  by  BIT/ETE.  However,  for  any  higher 
level  of  analysis,  the  failure  rate  tested  is  merely  the  failure  rate  of  the  funetions 
or  equipments  that  are  monitored  by  BIT/ETE. 

TT  can  also  be  measured  without  using  failure  rates.  Instead  of  using  failure 
rate,  TT  can  be  measured  as  a  ratio  of  something  other  than  failure  rate  suoh  as 
the  number  signals  tested  relative  to  the  total  number  of  signals.  This  method, 
and  methods  similar  to  it,  (i.e.,  number  of  functions  tested,  number  of  units  tested, 
etc...)  is  by  far  the  most  inaccurate  measure  sinoe  it  does  not  aooount  for  the 
relative  frequency  of  failure  occurrences. 

S.  1.1.5  Analysis  Method  for  Fault  Isolation  Resolution  (FIR(L)) 

Analysis  of  FIR(L)  can  also  be  accomplished  by  using  a  ratio  of  failure  rates. 
For  this  case,  there  will  be  a  set  of  ratios  used  due  to  the  way  FIR(L)  is  defined. 
Thus  given  FIR(L)  expressed  as< 

FIRdj)  *  Pj 

fir(l2)  =p2 
fir(l3)  sp3 


where  Pj  <  P2  <  Pg  s  1 

then  the  model  for  analyzing  FIR(Lk)  is* 


FIR(Lk)  is, 


£  <VW 

j  «  1  J  for  k  =  1,  2,  3 


where*  NR-  the  number  of  RIs  in  the  system 

*LKj  -  failure  rate  of  the  RI  associated  with  the  fault 
isolation  resolution  level 


*LKj  *  (*j-*UDj>  for  ttU  J 
-  failure  rate  of  the  Jth  ri 

\  ii. 

aUDj  -  failure  rate  of  the  jin  RI  that  Is  undetectable  by  any  means 
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In  order  to  determine  the  values  for  the  above  model,  thorough  knowledge 
of  the  BIY/ETE  fault  isolation  test  philosophy  is  required.  Not  only  does  this 
require  information  on  the  BIT/ETE  test  capabilities  and  the  test  oiroultry,  it 
also  requires  a  thorough  understanding  on  how  the  tests  are  documented  (e.g. , 
software  documentation  and  maintenance  manuals  documentation),  slnoe  FIR(L) 
is  normally  dependent  upon  human  interaction. 

5. 1, 1.6  Analysis  Method  for  Fraction  of  Faults  Isolated  (FFI) 

As  noted  In  Section  3, 2. 10  FFI  Is  greater  than  or  equal  to  FIRflfc),  where 
is  for  the  maximum  speolfled  fault  Isolation  group  site.  That  la 
FFI  »  FIR(Lk) 

whom 

“  maximum  specified  fault  isolation  group  size 

FFI  can  be  expressed  ast 
N« 


FFI  * 


"1C- 

i=l 


where) 

N0  -  number  of  components 

Xj  -  failure  rate  of  the  i^  component 

Xjj  -  failure  rate  of  t^v  component  that  is  isolated  by  BIT/ETE 


Determination  of  the  failure  rate  associated  with  the  numerator  and  denomi¬ 
nator  of  FFI  requires  the  same  Information  as  is  required  for  FIR(L)  (refer  to 
Section  6. 1. 1. 6).  However,  for  FFI  knowledge  of  the  fault  isolation  group  sites 
(i.e. ,  fault  isolation  ambiguity)  is  not  necessary. 

6. 1.1.7  Analysis  Method  for  Fraction  of  Erroneous  Fault  Isolation  Results  (FKFI) 
FEFI  was  defined  as  the  fraction  of  erroneous  fault  isolation  results.  Be¬ 
cause  of  the  definition,  FEFI  cannot  be  analyzed  until  the  uystom  or  equipment 
is  operable  or  the  BIT/ETE  fault  isolation  function  is  operable.  Thus,  FEFI 
can  only  be  determined  through  testing, 

The  major  reason  for  the  inability  to  analyte  FEFI  is  due  to  the  philosophy 
of  hardware  design.  The  fault  isolation  test  routines  are  usually  written  assum¬ 
ing  that  they  are  correct.  Thus,  it  la  not  possible  to  determine  whether  ft  fault 
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isolation  result  is  erroneous  until  actual  testing  is  performed  on  it*  However, 
there  is  one  possibility  of  analysing  FEFI  before  testing.  If  the  test  routines 
are  thoroughly  defined,  then  with  the  aid  of  an  FMEA,  the  effect  due  to  a  single 
failure  can  bn  analyzed  and  erroneous  fault  isolation  results  may  be  flagged  out 
before  they  ooour,  This  is  a  very  tedious  task  and  it  is  unlikely  that  a  task  of 
this  type  will  be  performed. 

5.1.2  Time  Dependent  FOMs 

Fg  and  Tr  are  BIT/ETE  FOMs  that  are  dependent  on  timo  only.  Thus,  the 
analysis  of  those  FOMs  requires  only  a  time  line  analysis.  Time  line  analysis 
consists  of  the  measurement  of  a  sequence  of  events  using  one  of  the  following 
methods  of  measurement) 

1.  actual  time  measurements  (i.e, ,  time  study) 

2.  tables  of  standard  elemental  times 

3.  engineering  judgment  (or  analysis) 

Timo  line  analysis  using  actual  time  measurements  consists  of  measuring 
(with  u  stopwatoh  or  something  similar)  how  tong  it  takes  to  perform  a  sequence 
of  events.  Time  lino  analysis  using  standard  elemental  time  consists  of  recon¬ 
structing  a  sequence  of  events  using  standard  times  (o.g. ,  MIL-HDBK-472, 
RADC-TR- 70-89,  or  RADC-TR-78-109)  for  the  various  actions  that  comprise 
the  sequence  of  ovonts.  The  final  method  for  measuring  the  time  required  to 
porform  a  sequence  of  events  is  to  uso  engineering  Judgment  or  analysis  to 
analytically  determine  how  long  It  takes  to  porform  the  tasks, 

To  perform  a  time  line  analysis,  the  following  steps  are  required) 

1.  identification  of  the  sequence  of  events  that  are  to  bo  measured 

2.  determination  of  the  time  required  to  perform  each  unique  stop 
using  one  of  the  three  methods  presented 

3.  determination  of  the  timo  to  perform  the  identified  sequence  of  ovonts 

Identification  of  the  sequence  of  events  that  urc  to  be  measured  is  tho  most 

important  stop,  If  the  proper  actions  required  are  not  identified  correctly, 
then  the  time  line  analysis  is  invalid. 

The  following  subsections  discuss  the  application  of  the  time  line  analysis 
technique  for  the  associated  FOMs. 
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fi.  1.2. 1  Analysis  Method  for  Frequency  of  BIT/ETE  Executions  (F^) 

Since  moat  periodic  BIT/ETE  test  routines  arc  normally  computer  con¬ 
trolled,  oan  be  dotormined  with  respect  to  the  clock  rate  of  the  computer 
CPU  (Central  Processing  Unit),  that  controls  the  tost. 

As  noted  in  the  definition  of  F^,  F^  is  defined  ass 
MIN  (FB^>  or  MAX  (Toyolo  )  for  all  i 


whores 

fh 

F«  -  oycling  rate  of  the  i  continuous  BIT/ETE  test  routine 
“i 

Tcyclo  ”  °y°llnS  ("  _1)  of  the  i^  continuous  BIT/ETE  tost 

1  routine  1 

Measurement  of  each  cycle  time,  Tcyc^c  ,  can  bo  determined  analytically  byi 

-i.  .  (tho  number  of  CPU  clock  cycles  between  starts  of  the  ith  BIT/ETE  test  routine) 

cyclop  ~  (CPIT clock  rate)  ~ 

Thus,  the  frequency  of  execution  of  the  i^  BIT/ETE  tost  routine  is; 

11  Bj  “  ^oycloj* 

6. 1,2,2  Analysis  Method  for  Mean  BIT/ETE  Punning  Time  (Tfi) 

Ah  stated  in  the  definition  of  (Section  3.2.0),  there  are  sovoral  defini¬ 
tions  for  Tg.  Once  the  definition  of  Tfi  has  boon  determined  (1.  o, ,  for  what 
tests,  with  or  without  a  fault,  etc. ..)  analysis  of  the  moan  BIT/ETE  running 
time  is  accomplished  by  using  the  following  model) 


T  v*  mi 

1 B 


whore) 


Nj}  -  tho  number  of  BIT/ETE  tost  routines  included  in  T^ 

Tn  ••  the  running  time  for  tho  ith  BIT/ETE  tost  routine 
Bi 
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is  determined  by  the  following: 


it. 

t  {number  of  instruction  stops  cycled  by  the  1  BIT/jCTK  tost  routine: 
iB1  (computer  instruction  cycling  rate) 

The  number  of  instruction  steps  cycled  is  dependent  upon  the  definition  of 
Tfi.  For  example,  if  Tg  were  defined  for  "with  a  fault  found"  and  the  BIT/ETE 
test  routines  wero  organised  such  that  the  test  "loops  on  an  error"  until  a  oer- 
tain  threshold  level  has  boon  obtained,  then  each  Tj^  must  account  for  the  reit¬ 
eration  of  the  test  steps  that  are  in  tho  loop. 

5.1,8  Failure  Rate  and  Time  Dependent  FOMb 

FOMb  that  are  failure  rate  and  time  dependent  (i,  e. ,  TpD,  T^j,  MTTR, 
MTTRg/p)  ure  normally  FOMs  whose  time  average  is  dependent  upon  the  rate 
of  occurrence  (1.  o. ,  failure  rate).  These  FOMs  can  be  analyzed  using  current 
anulysis  techniques  similar  to  MIL-HDBK-472,  procedure  2  and  RADC-TR-78- 
1(59,  a  failure  rate  weighted  average  of  times, 

As  was  true  for  tho  failure  rate  dependent  FOMs  (Stxstion  5. 1. 1)  these  FOMs 
can  also  be  analyzed  at  various  levels  (i.e, »  equipment,  unit,  component,  oto.). 
The  general  model  for  analyzing  FOMs  of  this  type  is  very  similar  to  the 
models  of  Tabic  10.  With  slight  modification  of  the  models  in  Section  5. 1. 1, 
tho  general  model  for  u  component  level  anulysis  can  bo  expressed  as: 


average  tlmo 


Nc 

£  v 


it 

(tj,  the  time  for  the  i  component) 


Nc 

L  \ 

l^t  1 


where: 

tj  -  Is  the  time  to  perform  the  action  defined  by  tho  FOM  for  tho  i*1' 
component,  (tj  is  determined  by  time  lino  analysis) 

U. 

Xj-  failure  rate  of  the  t  component 

The  same  philosophy  holds  true  for  failure  rate  and  time  dependent  FOMn 
as  did  tor  tho  failure  rate  dependent  FOMs.  The  accuracy  of  the  analysis  is 
dependent  upon  tho  lovol  at  which  the  analysis  is  performed. 
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Straight  methods  for  determining  MTTR  and  MTTRByK  are  readily  avail¬ 
able  (e.g,,  MIL-HDBK-472  and  RADC-TR-78-1G9)  and  require  no  further 
discussion  here.  These  same  techniques  are  also  applicable  to  and  TFJ, 
differing  only  in  the  representation  of  the  synthesised  time,  t^.  For  Mean 
Fault  Detection  Time,  T™,  each  tj  represents  the  time  necessary  to  deteot 
and  indicate  a  fault  in  the  1th  component.  For  the  Mean  Fault  Isolation  Time, 

Tpj,  tj  represents  the  time  necessary  to  perform  the  fault  isolation  process  for 
a  fault  in  the  Ith  component. 

Tho  subdivision  of  the  failure  ruto  for  the  numerator  and  denominator  of  the 
defined  FOM  is  done  tho  same  way  it  is  done  for  the  failure  rate  dependent  FOMs 
(refer  to  Section  5. 1. 1). 

5.1.4  Analysis  Method  for  Maintenance  Personnel  Skill  Level  (MPSL) 

Analysis  of  MPSL  is  dependent  upon  the  way  it  has  been  defined.  As  noted 
in  Seation  3.2. 12,  MPSL  can  be  defined  as  an  average  or  a  minimum.  In  most 
cases,  it  is  defined  as  a  minimum  since  it  will  usually  denote  the  skill  level  that 
will  maintain  the  system.  Analysis  methods  for  both  definitions  are  pre¬ 
sented  hero. 

If  MPSL  is  defined  as  the  minimum  skill  level  required  to  maintain  a  system, 
then  the  following  model  holds  true. 

MPSL  »  MIN  (MPSLi)  for  all  i 


whorot 

a,  |* 

MPSLj  -  tho  skill  level  required  to  perform  maintenance  for  the  i  failure 


Normally  if  MPSL  is  defined  us  a  minimum,  it  will  nlso  have  a  threshold 
level  specified  with  it  (e.g, ,  <p  percent  of  all  faults  will  bo  maintainable  by  a 
specified  skill  level),  if  this  is  true,  then  MPSL  can  bo  analyzed  as  follows: 


"c 

a  X 

Sp 

Si*1 


x  100  a 


where: 

X,,t  -  failure  rate  of  the  l^'  component  that  Is  maintainable  by  a  skill  level 

1  Ji 

1  less  than  us  equal  to  tho  specified  minimum 


♦in  order  to  satisfy  tho  requirements  imposed  this  relationship  must  bo  true. 


5.2  FORMAL  DEMONSTRATION  OF  FOMS 

One  way  to  assess  the  dogree  to  which  certain  FOM  requirements  have  been 
met  is  through  a  formal  demonstration  tost.  Such  a  tost  is  usually  of  a  statisti¬ 
cal  nature,  i.e,,  certain  random  quantities  containing  Information  about  tho  true 
value  of  the  FOM  under  consideration  arc  observed  and  evaluated  (for  example, 
total  number  of  failures  ih  a  given  tost  time  for  n  reliability  demonstration  test). 

Formal  demonstration  testB  already  exist  for  tho  FOMs  MTBFB//K, 
MTTRgyg,  and  MTTR,  and  are  described  in  military  standards  (M1L-STD-471 
and  MIL-STD-781).  These  standards  also  apply  to  A  and  A^j,,  indirectly.  For¬ 
mal  demonstration  tests  for  FFD,  FFI,  FK FI,  and  FIR  are  described  in  Seotions 
6.  i,  1  and  5.2.2  The  remaining  FOMs  (i.o. ,  FFA,  FFS1,  TpI),  Tfi,  F^,  T  , 
and  MPSL)  do  not  lend  thomsolvos  to  statistical  demonstration  procedures*  and 
must  bo  assessed  using  analytical  moans  or  on  the  basis  of  field  data  (for  a  dis¬ 
cussion  of  those  concepts,  see  Section  5.3). 

5*2,1  Demonstration  of  Certain  FOMs  Expressed  as  Fractions 
5. 2. 1 . 1  Mathematical  Discussion 

It  is  clour  from  tho  analytical  definitions  of  FFD,  FFI  and  FEFI  that  these 
FOMs  may  be  Interpreted  ns  probabilities  that  certain  events  occur.  It  is,  thus, 
appropriate  to  develop  demonstration  tost  plans  for  those  FOMs  based  on  the 
binomial  distribution.  Statistical  demonstration  testH  bus  d  on  the  binomial 
distribution  will  be  developed  In  this  section.  The  applicability  of  those  methods 
to  spedflo  FOMs  (FFD,  FFI,  and  FEFI)  will  bo  discussed  and  illustrated  with 
examples  in  tho  next  four  sections. 

Denote  by  8  u  random  variable  (also  referred  to  as  a  test  statistic)  which 
huB  a  binomial  distribution  with  parameters  n  (a  positive  integer)  and  p  (the 
probability  of  "success”).  That  is, 

PjV  k!  &  (k)pV“P>n~k  (5.2.  1.1.1) 

for  k  ~  0,  1,  ....  n.  For  fixed  n,  tho  procedure  for  testing  the  simple  hypothesis 
(Ho)  p  pQ  against  the  simple  alternative  (11  j)  p  :  p^,  Pj  <  p()  is  to 

Reject  Ho  if  Sns  k  (hence  aecopt  H^)* 

Accept  Hq  if  k  +  1  (hence  reject  Hj), 

(Tho  numbor  k  is  called  the  aoeept/rojoet  criterion).  Here  po  may  be  thought  of 
as  tho  "design  goal"  or  speoifiod  value  of  the  FOM  under  consideration,  and  Pj 
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is  the  "Minimum  acceptable"  value  of  the  FOM  under  consideration.  It  hns  been 
taoltly  assumed  here  that  values  of  the  FOM  closer  to  1. 0  are  the  more  desirable 
valueB,  Therefore,  in  particular,  we  will  be  demonstrating  1-FEF1  instead  of 
FEFI.  The  parameter  "n"  is  the  sample  size  on  which  the  test  statistic  Sn  is 
based.  The  statistic  SR  is  the  total  number  of  "successes"  in  n  Independent  ex¬ 
periments.  For  example,  for  FFD  n  sample  faults  would  be  Inserted  or  simu¬ 
lated  in  the  equipment  undergoing  demonstration  testing.  In  this  onse,  Sn  would 
be  the  number  of  the  n  sample  faults  subsequently  detected  with  BIT/ETE.  The 
statistic  Sn  will  thus  have  a  binomial  distribution  given  by  equation  (6, 2, 1. 1. 1) 
with  p  equal  to  the  actual  probability  of  fault  detection,  l.e. ,  the  actual  valuo  of 
the  FOM  FFD.  The  value  k  in  (*)  is  determined  for  a  given  n  and  pQ  by  soleotlng 
the  produoer  risk  a,  or,  equivalently,  k  is  determined  for  fixed  n  and  pj,  by 
selecting  the  consumer  risk  p.  The  value  <v  is  the  probability  that  the  demon¬ 
stration  test  is  failed  given  that  the  design  goal  has  actually  been  met.  The 
value  fi  is  the  probability  that  the  demonstration  tost  is  passed  given  that  only 
the  minimum  acceptable  has  boon  achieved.  Mathematically, 


(B.  2. 1,1.2) 


(5.2. 1.1.3) 


Since  (5.2. 1, 1. 1)  defines  a  discrete  probability  distribution  function,  preselected 
va  lues  of  a  and  0  will  not  exactly  bo  achievable.  However,  (6. 2. 1. 1, 3)  or 
(5. 2. 1,  J ,  2)  will  determine  a  valuo  of  k  which  provides  a  consumer  or  produoer 
risk  closest  to  the  preselected  values  of  consumer  and  produoer  risk,  respectively. 
It  oan  be  shown  (reference  31  that  the  hypothesis  test  (*)  is  optimal  in  the  sense 
that  for  a  fixed  value  of  at,  the  resulting  k  yields  the  smallest  possible  0  and 
vice  versa. 

To  aid  in  the  design  of  demonstration  tests,  Appendix  A  contains  listings 
of  the  exact  probability  (using  the  binomial  distribution)  of  accepting  H0 
(i.e, ,  passing  the  test)  as  a  function  of  the  true  value  of  p  for  fixed  pairs 
of  n  and  k.  The  values  n  »  20,  25,  30,  35,  40,  45.  50,  80,  70,  80,  00,  100 

were  ohosen  along  with  (for  each  n)  k  *  0,80n,  0.80n  +  1 . n-1.  Denoting 

by  y(P)  the  probability  of  accepting  H0  as  a  function  of  the  true  value  of  p 


(i.  e.  ,  y  (p)  represents  the  entry  in  Appendix  A  oor responding  to  the  probabil¬ 
ity  of  passing  the  test),  the  exact  values  of  a  and  p  may  be' read  directly  (or 
interpolated)  from  the  tables  for  fixed  (n,  k).  by 
Ml-  (p0) 

8  *  Y <Pi>* 


<5.2,1. 1.4) 


The  use  of  Appendix  A  will  be  lllustraBted  for  designing  test  plans  for 
the  FOMs  FFD ,  FFI ,  and  FEFI  in  the  next  sections.  Often,  specific  values 
for  Po>  Pi*  a  and  e  Will  be  ohosen  and  it  will,  thuB,  be  necessary  fo  find  the 
sample  size  n  and  the  pass /fail  value  k  necessary  to  complete  the  definition 
of  the  demonstration  test.  This  could  bo  accomplished  by  searohing  the 
table  for  the  desired  combination  (n,  k)  which  would  yield  a  «1-y(P0)  and 
8  M  y(Pi).  A  more  efficient  method  (though  only  approximate  in  nature) 
relies  on  using  the  normal  distribution  approximation  to  tho  binomial  distri¬ 
bution  (reference  3) .  Denoting  by  Zt  the  t*h  quantile  of  the  standard  nor¬ 
mal  distribution  (e.g.,  Zq  95  *  1.045)  the  approximate  valuo  of  n  and  k  for 
a  given  set  of  values  p0.  Pi»  a,  8  are  given  by  (see  roforenco  1), 


_  '  2 

lzi-/j'/pi(1-pi)  -  z /po(i-p0y| 

n  at  <  - - *■-— £1-.? - 2-1  (5.2.1. 1.0) 

w 

and 

k  u  npj+  Zlw^np^(iHpJ.  (5. 2. 1.1. 7) 

It  should  be  emphasised  that  these  values  on  n  nnd  k  are  approximations. 
Having  computed  n  and  k  from  (5,2,1. 1,6)  and  (5.2. 1.1.7),  this  pair  (n,  k) 
can  be  used  as  a  "starting  point"  In  searohing  table  5. 2. 1.1.1  for  better 
value  of  n  and  k  (i.o. ,  better  in  sense  that  they,  In  conjunction  with  the 
desired  values  p0  and  pi  yield  values  of  producer  and  consumer  risk  closer 
to  the  preselected  values  a  and  0,  respectively. 


Appendix  A.  2  gives  step-by-step  procedures  for  determining  demonstra¬ 
tion  tests  using  the  tables  in  Appendix  A,  Two  important  cases  are  covered: 
in  case  1,  n,  p^  and  0  are  known  and  pQ,  k  are  to  be  determined  assuming 
a“  In  case2,  p,  pi,  a  ,  and  pQ  are  known  and  n,  k  are  to  be  determined. 

5, 2. 1.2  Demonstration  of  Eraotion  of  Faults  Detected  (FFD) 

As  seen  in  Section  3.2,  FFD  can  be  either:  1)  the  probability  that  any 
fault  is  detected  by  BIT/ETE  or  2)  the  probability  that  any  detected  fault 
is  deteoted  by  BIT/ETE,  In  either  case,  it  will  be  necessary  (for  statistical 
demonstration  procedures)  to  insert  or  simulate  a  number  (say  N)  of  faults 
at  random  in  the  system  undergoing  demonstration  testing.  These  faults 
should  be  distributed  in  a  manner  such  that,  on  the  average,  portions  of 
the  equipment  possessing  higher  percentages  of  the  total  failure  rate  will 
possess  a  proportionately  high  percentage  of  the  total  number  N  of  faults 
(see  Section  5.4  for  further  discussion  of  fault  seeding).  In  either  onse 
(case  1  or  2),  the  test  statistic  will  be  the  total  number  of  the  N  "seeded" 
faults  deteoted  with  BIT/ETE,  In  case  1,  the  sample  Bize  n  will  simply  be 
N.  However,  in  case  2,  if  j  of  the  seeded  faults  are  not  deteotabte  by  any 
means,  then  the  sample  size  n  will  be  N-J. 

In  general,  the  magnitude  of  n  will  depend  on  the  minimum  acceptable 
vnlue  of  FFD  (p^),  the  consumer  risk  (0),  the  design  onpablllttes  (possible 
values  of  pQ),  and  the  producer  risk  (a).  For  example,  for  values  of  pQ  and 
P|  very  close  together ,  large  values  of  n  will  be  necessary  for  reasonable 
l’iaks  a  and  p.  Also,  if  pj,  p^,  and  n  are  fixed,  it  may  not  be  possible 
to  design  a  vnlue  p0  high  enough  to  allow  for  a  reasonable  value  of  n  (soe 
examplo  1).  Therefore,  care  should  be  taken  in  designing  a  demonstration 
test  for  FFD  (and  the  other  FOMs  in  this  seotion)  such  that  reasonable 
levels  of  risks  «  and  p  are  possible.  Often,  from  specifications,  it  will  be 
known  ahead  of  time  what  the  rough  values  of  a,  P,  p0,  and  will  be. 

In  tills  case,  (5.2. 1.1.6)  and  (5.2. 1.1.7)  can  be  used  to  determine  approxi¬ 
mate  values  of  n  (sample  size)  and  k  (pass /fail  criterion)  and  thon  Appen¬ 
dix  A  can  bo  used  to  "zero  in"  more  on  a  combination  of  n  and  k  which 
yields  exact  consumer  and  producer  risks  closer  to  the  preselected  a  and  p. 

The  preceding  conoopts  are  illustrated  below  with  three  examples.  These 
examples  alBo  serve,  with  minor  modification,  to  illustrate  the  demonstration 
of  FFI  and  FEFI. 


Example  1.  The  consumer  specifies  his  risks  as  /3  -  0, 95  and  the  minimum 
acceptable  value  of  FFD  (  fox’  nil  faults)  as  pi  «=  0,98.  The  consumer  further  speci¬ 
fies  thut  the  demonstration  test  will  be  based  upon  a  sample  size  of  N  =  n  45 
faults  (see  Appendix  A.  2,  case  1).  This  is  sufficient  information  to  determine  the 
aooept/roject  criterion  (or  simply  "test  criterion")  k.  Searching  Appendix  A  with 
n  u  45  and  pi  e  o,  98  for  various  values  of  k,  it  is  found  that  the  smallest  possible 
value  of  fi  (with  no  45,  p^  »  o. 98)  is  0, 4029  for  k  a  44,  Supposing  that  (3a  0, 4029 
Instead  of  0.05  is  acceptable,  let  us  determine  what  tile  design  goal  pQ  must  be. 
Soanntng  the  table  with  n  a  46,  k  a  44,  it  is  seen  that  p0  must  be  larger  than  0, 995 
to  aohievo  an  tv  level  loss  than  0, 20  (i.  e, ,  it  is  seen  that  1  -  y  (  9.  995)  «  0. 20). 

If  ae  0.20  is  acceptable  to  the  producer,  and  if  p0  «  0.995  is  a  realistic  design 
goal,  then  the  tost  which  most  olosoly  matchos  the  original  specifications  of  the 
oonsumer  is: 

Producer  passes  (HQ  accepted)  if 

#  of  faults  detected  *  &  let  1 «  45 

Produoor  fails  (H^  accepted)  if 

#  of  faults  doteotod  ■  S4e  ss  44 

The  exact  values  of  at  and  p  for  this  test  uro  0.20  and  0.4029,  respectively, 

Notice  that  not  only  was  it  impossible  to  aohievo  on  exact  0  oloser  than  0. 4029 
to  0. 05  (for  n  «  45,  p^  «  0.98),  but  the  test  oritorion  implies  tixat  all  45  faults 
must  be  detoctod  in  order  to  pass  tlxe  test.  Thus,  although  the  test  (i.  0. ,  tixat 
all  45  faultfl  bo  detected  in  order  for  the  producer  to  pass)  seems  vary  strict, 
tlxe  0  level  indloates  exoessivo  risk  on  tlxe  part  of  tlxe  oonsumer. 

Example  2.  The  oonsumor  speolfios  his  risk  os  0  «  0. 10  and  tho  minimum 
aeoeptable  value  of  FFD  (for  all  faults)  as  0. 945.  He  further  specifies  tixat  the 
demonstration  test  will  be  based  upon  a  sample  of  size  N  ■=>  n  «=  70  (see  Appendix 
A.  2,  oase  1).  Looking  In  Appendix  A,  it  is  seen  that  for  the  ease  n  -  70,  k  =  88, 
y(0.945)  1=  0.0967  whioh  is  the  exact  p ,  Thxxs,  the  demonstration  tost  is 

Producer  passes  (llQ  aaoeptod)  If 

#  of  faults  deteotod  *  fe  k+1  69 

Producer  fails  (II j  aooepted)  if 

fl  of  faults  dotoctcd  ■  S?0  s  68 

Since  pf)  has  not  boon  spoolfiod  hero,  tlxe  producer  must  oxaxxxlne  tho  table  for 
n  '  -  08  to  find  tho  valuo  p0  to  which  I10  eon  design  in  order  to  achieve  his  desired 
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level  of  risk,  a.  If  a  ~  0. 10  is  acceptable,  it  is  soon  from  the  table  (for  n  =  70, 
k  =  (58)  tliat  1  -  y (0. 99)  «  0. 1553  and  1  -  y (0. 995)  »  0. 0483.  Simple  linear  inter¬ 
polation  yields  p  «  0.9920  as  tho  design  goal  for  FFD, 

Example  3.  Assuxnc  a  «  p  •  0. 10,  pi  «s  0, 80,  p0  »  0, 90  (see  Appendix  A,  2, 
oase  2).  To  complete  the  definition  of  a  demonstration  tost,  n  and  k  must  be  deter¬ 
mined.  From  5. 2.1.1.  0  and  5. 2. 1.1. 7  it  is  seen  that  (since  Z^  -1,282=  -ZQ  QQ) 


n  a 
&  81 


(1.282  V(0. 80)(0. 20)  +  1.282  )/( 0. 90)(0.  10)) ! 

(0.90-0. 80)2 


and 


k  «  (81)(0, 80)  +  1.282  V^81)(0.80)(0.20) 

W  09 

The  ease  n  =  81  was  not  inoludod  in  Appondix  A  but  considering  the  oases  in 
Table  5. 2. 1. 1  for  n  «  80,  it  is  seen  that  for  n  <->  80,  k  «  08,  a  «  1  -  y  (0, 90)  « 

0. 1004  and  d  «  y  (0, 80)  m  o.  lOOO,  Thus,  since  these  exact  value n  arc  extremely 
oloso  to  the  prosoloctod  values  of  a  **  /l «  0. 10,  tho  tost  is  thus! 

Producer  passes  (1IQ  aooopted)  If 

#  of  faults  dotootod  n  S^Q^k+l  *»  (39 
Producer  fails  (11^  accepted)  if 

#  of  faults  detected  ~  S^0  ssG8, 

From  this  example  It  is  soon  that  equations  5. 2. 1. 1. 0  and  5. 2, 1. 1. 7  can  yield 
values  of  n  and  k  which  in  turn  yield  values  of  a  and  (2  quite  close  to  the  preselec¬ 
ted  values  of  a  and  (1.  in  this  example,  howover,  the  value  of  n  computed  in 
5.  2. 1, 1,0  satisfied 

np+  fe  5  (6.2. 1.2,1) 

whore 

P*  --  minjp0,  l-po,  p1,  l-p1J. 

It  can  be  shown  (reforenoo  3)  that  condition  5.2, 1,2.1  is  sufficient  for  the  normal 
distribution  approximation  used  In  the  derivation  of  5. 2. 1.1,6  and  5. 2„  1 . 1. 7  to  bo 
a  good  approximation.  Values  of  n  and  k  computed  from  5. 2. 1 . 1. 0  and  5.  2. 1 , 1, 7 
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which  do  not  satisfy  5. 2. 1.2. 1  cannot  be  expected  to  be  reliable.  However,  such 
values  will  still  provide  a  starting  point  for  determining  values  n  and  k  which  aro 
acceptable , 

S.  2. 1. 2  Demonstration  of  Fraction  of  Faults  Isolated  FFI 

To  demonstrate  FFI,  N  faults  are  inserted  or  simulated  randomly  in  the 
system  undergoing  the  demonstration  tost,  The  sample  size  n  (see  Section 
6.2, 1, 1>  is  n-J  whore  J  is  the  number  of  tin  N  faults  which  aro  not  detoctable 
by  any  means.  'Iho  test  statistic  Sn  is  simply  the  number  of  the  N-j  BIT  detected 
faults  which  are  isolated  to  the  replacement  lcvol  specified  by  the  maintenance 
cone  opt.  Soo  Section  0. 2. 1. 2  for  examples  of  how  to  design  specific  demonstra¬ 
tion  tests.  If  random  faults  occur  In  addition  to  the  seeded  faults,  the  sample 
size  is  adjusted  accordingly  (see  Section  5. 2. 1,2,  for  exnmple). 

6.2, 1.3  Demonstration  of  Fraction  of  Erroneous  Fault  Isolation  (FEFI)  Results 
Since  FEFI  is  the  probability  of  erroneous  fault  isolation,  it  will  bo  more 
convenient  to  demonstrate  1-FEF1.  To  accomplish  tills,  N  faults  aro  inserted  or 
simulated  randomly  in  iho  system  undergoing  the  demonstration  tost.  The  sample 
size  n  (soo  Section  6. 2. 1, 1)  Is  N-l  whoro  i  Is  the  number  of  the  PT  seeded  faults 
which  cannot  bo  detected  by  BIT.  The  tost  statistic  Sn  Is  then  the  numbor  of  the 
N-l  faults  which  are  correctly  Isolated  by  BIT,  Sec  Section  6. 2. 1. 2  for  examples 
of  how  to  design  specific  demonstration  tests.  If  random  faults  occur,  in  addi¬ 
tion  to  the  N  seeded  faults,  the  sample  size  n®  N-l  is  adjusted  by  adding  in  the 
numbor  of  additional  random  faults  detected  by  JUT. 

5,2,2  Demonstration  Tests  for  Fault  Isolation  Resolution  (FIH(L)) 

The  FOM  FlR(li)  may  bo  interpreted  us  the  probability  that  a  fault  (do  loo  tod 
by  BIT)  will  be  Isolated  by  BIT  to  L  or  less  replaceable  items  (Rls)  (F1R(L))  is  a 
generalization  of  FFI).  This  FOM  is  often  specified  nt  more  than  one  level  1..  That 
is,  there  may  bo  k  levels  <  . . .  <  1^  with  corresponding  specified  values 

(F1R(L1)  <  FIR(l<2)  < . . .  <  FIK(L^),  Equivalently,  Uie  probabilities 

Pl  -  FIR  (Lj) 

p2  «  FIR  (L2)  -  FIR  (Lj)  (6. 2. 2. 1) 

» 

* 

Pk  “  FIR  (Lk)  -  FIR  <lw) 
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may  bo  apeoified.  It  Is  obvious  from  (5. 2. 2. 1)  that  a  given  Bet  of  Pp . . ,  ,pk 
will  uniquely  determine  FIR  (L^),  FIR  (L2)» . . .  *  FIR  (Lk)  and  conversely.  Tho 
probabilities  p^*.. .  «Pk  are  thus  interpreted  us: 

Pj  -  the  probability  of  isolating  a  fault  to  s  L  RIs 

p2  »  the  probability  of  isolating  a  fault  to  >  Lj  RIs  and  si  RIs, 


Pk  ■  the  probability  of  isolating  a  fault  to  >  RIs  and  a  Lk  RIs. 

Therefore,  pp* ,  .pk  aro  probabilities  of  k  mutually  exolusivo  events.  By 
analogy  with  the  demonstration  tests  of  the  lost  section  based  on  the  binomial 
distribution,  the  demonstration  tests  for  fault  isolation  resolution  will  be  based 
on  the  multinomial  distribution. 

If  there  are  a  total  of  NL  RIs  in  the  system  undor  consideration,  then  it 
will  bo  convenient  to  take  Lk  -  NR  so  that  FIR  (Lk)  will  always  bo  1.  In  this 
ease, 


k 

Ipj  «  FIR(Lk)«l.  <5. 2. 2. 2) 

i»l 

To  design  a  demonstration  test  for  fault  isolation  resolution,  it  will  bo  neces¬ 
sary  to  know  both  the  minimum  acceptable  and  design  goal  values  for  tho  vootor 
of  probabilities  p  (pp  ...,Pk). 

The  demonstration  test  statistic  is  defined  as  follows;  Insert  or  simulate  N 
faults  at  random  into  the  system  undergoing  demonstration  testing  and  suppose  that 
n  (n  s  N)  of  thoso  faults  aro  detooted  by  BIT.  Let; 

Xj  -  the  number  of  the  n  faults  detected  by  BIT  whioh  are  Isolated  by 
BIT  to  a  Lx  RIs 

Xg  «  the  number  of  tho  n  faults  detooted  by  HT  which  arc  Isolated  by 
BIT  to  >  L^  RIs  and  a  L2  RIs 


-  the  numlKU’  of  ho  n  faults  detooted  by  BIT  which  aro  isolated  by 
HIT  >  Lk->1  RIs  and  a  RIs, 


71 


(5. 2, 2. 3) 


(5. 2. 2. 4) 


Then  the  test  statistic  is  defined  by 

k  ,  *  k-1  ,  * 

T  “  ^  Xiln(pi/pl  )  =  2  AlXi+  n  ) 

where 

Ai  -  ln(Pi/pi+)  -  In  (P^/p^),  Ipi.'  2,  ....  k-1 , 

with p'^»  ...I  pj|  denoting  the  design  goal  values  for  the  yector  of  probabilities 
p,  and  p1*.  . . .  p^*  denoting  the  minimum  acceptable  values  for  the  vector  of 
probabilities  p.  The  test  statistic  definod  by*f^.  2.2.$)  follows  from  the  usual 
likelihood  ratio  tost  for  testing  the  simple. hypothesis  (HQ)  that  p  *»  (p^,  ....  p^) 
against  the  simple  alternative  (Hj)  that  p  -  (p^*,  ...»  p^*)  and  the  test  based  on 
5. 2. 2. 3  is  optimal  in  the  sense  discussed  in  Section  5. 2. 2, 1  (reference  3).  The 
demonstration  test  is  thin 

Producer  fails  test  if  T  <  C  (5.2.2  5) 

Producer  passes  test  if  T  a  C 

Where  C,  the  pass/fail  criterion,  is  determined  by  selecting  either  the  producer 
risk  a  or  the  oonsumer  risk  0  (but  not  both  simultaneously).  Since  the  random 
veotor  (Xj  ,  Xfc)  has  n  multinomial  distribution  under  either  hypothesis,  that 
is ,  for  non-negative  integers 


x.i  •••.  X,,  with  2  x,  *»  n; 


i»l 


P )  X  J  n  Xjt  •  9  •  ft 


V 


n! 


(s.v) 


nPj 

i»i 


(5.2.2.  6) 


Where  Pj ■  pj  (i a  1,  2,  ...»  k)  under  HQ  and  =»  (i  =»  1,  2,  k)  under 
the  value  0  is  determined  by  either  one  of  the  following  equations : 


a  «  2 
A 


0  -  8 
B 


n!  .  k 

E —  nP;xi 

n  x. !  i-il 

m  1 

T^-.  L  •  Xj 

“fi' 


(6.3.2. 7) 


(6.2, 2.8) 
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where:  A  is  the  set  of  all  vectors  of  non-negative  integers  (x, ,  . . . ,  x. )  such 
that 

k 

X  x,  «  n  and  T  <  C;  D  is  the  set  of  all  vootors  of  non- 
i«l  1 

k 

negative  Integers  (xlt . . . . »  x^)  such  that  x{  «  n  and  T  a  c. 

As  with  the  binomial  ease  discussed  In  Section  5. 2. 1. 1,  exact  prespeolflod  values 
of  a  or  (3  will  not  be  attainable  due  to  the  discreteness  of  the  multinomial  dis¬ 
tribution,  Howevor,  a  value  of  C  can  be  determined  from  6, 2, 2. 7  or  8. 2,2,8 
which  yields  an  exact  value  of  a  or  p  closest  to  the  prospeoifted  value. 

Since  the  sample  sko  n  for  a  demonstration  test  of  fault  isolation  resolution 
will  ofton  be  largo,  the  pass  fail  value  C  can  be  determined  using  normal  distri- 

it 

button  approximations.  Denoting  by  as  the  t  quantile  of  tho  standard  normal 
distribution  (e.g,,  ZQ  “  1.282)  and  utilizing  tho  normal  approximation  in 
(5. 2. 2, 8)  the  value  of  C  is  approximately 

G«  C0+  nln  (P^/Pg)  (C.2.2.9) 

where 


C  «  n 
0 


a,  V*1  V>  -  AjA.p/ 

1  1  la  KJ  ask-1  1  J  1 


(5. 2.2.10) 

dene  rally,  oonditions  (5. 2. 2, 7)  and  (5.2.2. 8)  will  not  be  satisfied  simultaneously 
for  a  given  value  of  n.  However,  when  n  is  presumed  to  bo  largo,  the  normal 
approximation  may  bo  applied  to  both  <8. 2, 2, 7)  and  (8. 2. 2. 8)  to  yield  tho  value 
of  n  neoessary  to  satisfy  (8, 2, 2, 7)  and  (8. 2, 2, 8)  simultaneously.  Tho  required 
value  of  n  Is  approximately 


2  2 
n  “  (Zl-/3  V'>  Am*V  ) 


(8.2.2.11) 
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,  k-1 

/i  »  2  A.p  ' 

i  »  1  11 

A  k-1  * 

M  =  2  A.p. 
i«l  1  1 


*'  -  VjPi'pj' 
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/k-1 

*  /  v  2  *  *  *  * 

,-/maipi  <A-pi  >  “  ^  Wi  PJ 


(5.2.2.12) 

<5.2. 2.13) 


(5.2. 2.14) 

(5.2.2.15) 


Tho  results  (5. 2. 2. 9)  through  (5.2,2. 10)  are  dismissed  and  derived  In  reference  1 
These  results  will  now  bo  lllustratod  by  some  examples ,  Appendix  B  con¬ 
tains  several  demonstration  tost  plans  for  various  sample  sizes  and  doslgn  goal 
and  minimum  acceptable  valuos  for  p  for  tho  case  k  «  3  (i.o. ,  two  actual  levels 
of  fault  isolation  resolution,  ihe  third  being  L,}  -  N^).  In  Appendix  13  tho  pass/ 
fail  valuo  C  was  computed  using  (5. 2. 2. 10)  for  the  prospoclflcd  value  of  (i.  Then 
oxaot  value o  of  a  und  /l  wore  oomputod  using  (5. 2. 2. 7)  and  (5. 2, 2. 8)  for  the  C 
valuo  computed  from  (5. 2. 2. 10).  It  con  be  seen  that  tho  approximation  (5. 2. 2. 10) 
is  very  good  for  tho  sample  sizes  considered  and  thop  values  considered, 


Example:  Suppose  thoro  arc  two  lovols  (i.o, ,  k -  3)  of  fault  isolation  resolution 
under  consideration.  Spooifioally,  tho  levels  L^,  Lg  imd  tltc  minimum  aoooptablo 
and  design  goal  valuos  for  FIR  (Lj)  and  FIR  (L,,)  arc: 


Lj-  1,  L£»  3  (L3-  Nr) 

Design  Goal:  FIR  (1)  -  0,800  or  pl*  -  0. 80 
Fm  (3)  «  0,900  or  p2'  «  0. 10 
FIR  (Lg)  -»  1.000  or  p3'  •"  0.10 

Minimum 

Aoceptablo:  FIR  (1)  -1  0.700  or  Pj*  «  0, 70 
FIR  (3)  0.875  or  p2*  «  0.175 

FIR  (Lj) «  1. 000  or  p3*  »  0, 125 


Suppose  further  that  the  oonaumer  risk  /9  is  prespeoifled  os  0. 10,  If  n  faults 
are  detected  by  BIT  (out  of  the  original  N  Inserted  or  simulated  faults)  then  let 

«  tho  number  of  the  n  faults  isolated  by  BIT  to  1  RI 
Xg  «=  the  number  of  the  n  faults  isolated  by  BIT  to  >  1  RI  and  s  3  RIs 
Xg  ■  the  number  of  the  n  faults  isolated  by  BIT  to  >  3  RIs  and  si  Lg  RIs. 

From  Table  B-5  of  Appendix  B  (or  using  (5.2,2, 1),  (5. 2, 2, 3)  and  (5. 2. 2. 4))  it  is 
seen  that: 

T  «  0. 13353  Xj  -0. 55962  Xg  -0. 22314  Xg. 

Again  from  the  Toblo,  it  is  seen  that  for  n  «  50,  the  exact  value  of  ft  is  0.0889 
and  the  exact  value  of  tho  producer  risk  a  is  0.3557,  Iho  pass/fail  value  C  is 
0, 822G,  so  that  the  producer  passos  the  tost  (when  n  »  50)  if  T  &  0, 8226  and  foils 
otherwise.  If  tho  valuo  0.3557  is  too  large  for  a ,  then  a  larger  sample  size  n 
will  bo  required.  If  o  «  0. 10  is  desired,  then  die  required  n  may  bo  approximated 
uBlng  (5. 2, 2. 11).  The  oalouktion  is; 

*  t  2  ^  (  2 

n  Si  (1.282  ar  +  1,282 or )  / < #*  -ju  ) 
whore 

IX  »  (0.13353)  (0,80)  -  (0.55962)  (0. 10)  »  0.05086 
H*  “  (0.13353)  (0.70)  -  (0.56962)  (0,175)“  -0.00446 

a  m  ((0, 13353)2  (0.80)  (0.20)  +  (0.55962)2  (0.10)  (0.90)  + 

2(0, 13353)(0.05962)(0.80)(0. 10)] 1/2  =  0.20735 

or*  *  f  (0. 13353)2(0. 70)(0. 30)  +  (0.  S5962)2  (0. 175)  (0. 825)  + 

2(0. 13353)(0.  55962X0, 70)(0, 175)) 1/2  -  0.  25930 


The  value  n  >=  117  Is  not  included  in  the  Table  but  tor  n  »  110,  a  “  0, 1051  and 
li  «  0.0910,  Since  these  values  are  very  close  to  0. 10  the  test  for  n  «  110  may 
bo  used.  For  tho  case  n r*  110,  tho  pass /fall  value  C  is  0.0000  and  the  producer 
pusses  if  T  &  0, 0000  and  fails  otherwise.  Suppose  thati 

X1  -  84 

Xg  “  11 

Xg“  18 

is  obsorvod  in  the  tost  with  n  «  110. 

Thon,  T  ■  (0.10353)  (84)  -  0.85002(11)  -  0.22314(10)  «  1.7180  >  0,0006  so,  the 
producer  pusses. 

If  the  observed  results  are: 

Xj«  82 

X„  -  15 

V* 

then  T«  (0.13353)  (82)  -  (0.65902)  (15)  -  (0.22314)  (13)  «  -0. 34500  <  0.0000  so, 
the  produoor  fails. 

5, 2. 3  Fault  Seeding  Methods 

Tho  validity  of  the  demonstration  tests  for  FFD,  FFI,  FEF1,  and  Flit  is 
dependent  on  tho  ability  to  insert  or  simulate  f twits  in  the  particular  system  at 
random,  %  tho  term  "at  random",  it  is  meant  that  faults  are  inserted  or  simu¬ 
lated  In  such  a  manner  that  tuty  one  of  all  possible  faults  tit  at  can  occur  is  equally 
likely  to  bo  inaortod  or  simulated  on  any  Riven  fault  Insertion  or  simulation.  Of 
oourse,  to  do  this,  ait  exhaustive  list  containing  all  possible  fault  typos  is  re¬ 
quired.  From  suoh  a  list,  the  sample  faults  are  ohosen  at  random  and  Inserted  or 
simulated  In  the  system.  However,  an  exhaustive  list  in  usually  not  available 
and  would  be  extremely  costly  to  compile.  The  alternative  is  to  compile  tut  ab¬ 
breviated  list  which  provides  a  representative  sample  of  tho  exhaustive  list.  A 
random  sample  eon  thon  be  selected  from  this  fault  list  and  simulated  or  inserted 
in  tho  systom.  Suoh  an  abbreviated  list  con  be  obtained  from  techniques  suoh  us 
Appendix  A  to  MIL-STD-471A  which  provides  a  candidate  fault  list  for  maintain¬ 
ability  demonstration  tests. 

It  should  Ixi  pointed  out  that  when  more  than  one  FOM  requiring  fault  seeding 
is  being  demonstrated,  it  is  not  neocssury  to  seed  several  different  sets  of  faults. 
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Tho  same  sot  of  seeded  faults  may  be  used  In  eaoh  demonstration  tost  for  which 
tho  aoeded  faults  are  applicable.  Additional  fault  seeding  will  only  be  necessary 
if  some  demonstration  tests  require  larger  sample  sir, os  due  to  the  test 
parameters. 


V7 


6.0  GUIDELINES  FOR  SPECIFYING  BIT/E TE  FOMs 

This  section  provides  general  guidelines  for  the  selection  of  an  appropriate 
set  of  BIT/ETE  FOMs  that  should  be  specified  for  a  given  system/equipment. 
The  information  presented  here  is  provided  as  recommendations  only  for  select¬ 
ing  the  most  suitable  BIT/ETE  FOMs  to  speoify.  The  user  should  use  his  own 
discretion  when  determining  whloh  BIT/ETE  FOMs  will  finally  be  specified 
based  on  the  speoific  system/equipment  mission  and  objectives. 

This  section  is  broken  down  into  two  subsections.  The  first  subseotion 
provides  guidance  for  the  selection  of  BIT/ETE  FOMs  to  specify  t  given  various 
system/equipment  RMA  requirements.  The  second  subsection  oontains  guidanoe 
for  specifying  consistent  BIT/ETE  FOMs. 


6. 1  GUIDELINES  FOR  SELECTING  BIT/ETE  FOMs  TO  SPECIFY 

The  selection  process  for  an  appropriate  Bet  of  BIT/ETE  FOMs  to  specify 
Is  dependent  upon  the  system/equipment  In  question  and  the  requirements  or 
goals  that  are  desired.  In  other  words,  BIT/ETE  FOMs  should  be  specified 
according  to  the  most  Important  system  goals  that  are  desired  or  required.  This 
section  contains  a  methodology  that  oan  be  used  to  identify  candidate  BIT/ETE 
FOMs  to  specify,  based  on  the  desired  system  objectives  or  requirements. 

One  Important  point  should  be  noted*  the  BIT/ETE  FOMs  selected  will  most 
likely  supplement  the  RMA  FOMs  that  are  normally  specified  (e.g. ,  MTBF, 
MTTR,  and  A,  or  other  similar  FOMs),  With  this  In  mind,  it  Is  assumed  that 
at  least  two  of  the  three  RMA  FOMs  are  already  lnoluded  in  the  system  specifica¬ 
tion.  The  third  FOM,  whichever  it  may  be,  oan  be  easily  determined  using 
the  definition  of  availability 

_  MTBF  ) 

MTBF  +  MTTRA 

It  may  turn  up,  as  a  result  of  using  the  presented  guidelines,  that  the  RMA 
FOMs  that  are  already  a  part  of  the  system  specification  may  also  be  prime 
candidates  for  BIT/ETE  specification.  For  this  reason,  the  RMA  FOMs  that 
also  characterize  BIT/ETE  inherently,  remained  in  the  list  of  BIT/ETE  FOMs. 
Thus,  if  an  RMA  FOM  is  also  identified  as  a  prime  candidate  tor  BIT/ETE  speci¬ 
fication,  the  number  of  BIT/ETE  FOMs  required  can  be  kept  at  a  minimum. 

8.1.1  Development  of  BIT/ETE  FOM  Selection  Guidelines  Methodology 

The  matrix  shown  in  Figure  6  along  with  the  system/equipment  objectives 
enumerated  in  Table  17  are  used  to  select  candidate  BIT/ETE  FOMs.  The 
development  of  these  FOM  selection  guidelines  consisted  of  the  following  steps: 

1)  identify  the  various  system  objectives  that  are  related  to  BIT/BITE 

2)  correlate  the  system  objectives  with  the  BIT/ETE  FOMs  that  suit 
them  the  best 

3)  rank  the  BIT/ETE  FOMs  accordingly,  for  system  objectives  that  can 
be  characterized  by  more  than  one  BIT/ETE  FOM. 

The  first  step  of  the  BIT/ETE  specification  guidelines  development  was 
straightforward.  For  each  FOM  type  identified  in  Section  3,  a  system  ob¬ 
jective  or  reasons  for  specifying  that  particular  FOM  were  tabulated.  In 
other  words,  system  objectives  or  reasons  were  tabulated  depending  on  when 


with  the  Defined  BIT/ETE  FQMs 


TABLE  17.  SYSTEM/EQUIPMENT  RMA  AND  B1T/ETE  OBJECTIVES 


1.  A  high  confidence  of  continuous  performance  of  yystem/equipment 
requirements  is  desired  and  there  are  no  physical  constraints  (i.e. , 
redundancy  can  be  Implemented). 

2.  A  high  confidence  of  the  system's  correct  status  (1.  e. ,  operational  or 
down)  1b  needed  to  determine  prooeed/abort  decisions. 

3.  Minimum  downtime  due  to  failures  is  critical  and  physical  constraints 
restrict  the  use  of  redundancy, 

4.  Amount  of  spares  available  will  be  limited. 

6,  Minimum  maintenance  and  support  costs  are  desired. 

6.  Specific  skill  levels  will  be  required  to  maintain  the  systom/equipment, 

7.  Systom/oquipment  location  will  restrict  the  amount  of  maintenance  avail¬ 
able  (i.e.  •  remote  facilities). 

8.  BIT/ETE  failures  will  be  detrimental  to  system  reliability. 

9.  The  periodicity  and  length  of  on-line  testing  is  oritlcal  to  system  opera¬ 
tions  (e.  g. ,  in  redundant  systems  whore  switchover  times  must  ooour  as 
soon  as  possible), 

10.  The  amount  of  memory  for  resident  software  diagnostics  (l.c. ,  for  contin¬ 
uous  monitoring  capabilities)  is  limited. 

11.  Undetected  failures  are  hazardous  to  system  safety. 

12.  A  high  confidence  of  BIT/ETE  oporabllity  and  accuracy  is  desired. 


and  why  a  BIT/ETE  FOM  would  bo  specified.  For  oxamplo,  if  the  amount  of 
spares  available  for  a  system  is  limited,  then  FIR(L)  with  a  high  resolution 
(1.  o. ,  average  RI  group  size  relatively  low)  would  be  a  good  BIT/ETE  FOM  to 
specify. 

The  system  objectives  identified  were  general  in  order  to  maintain  an  over¬ 
all  system  applicability.  Table  17  summarlzca  the  various  system  objectives 
identified  that  pertain  to  BIT/ETE. 

The  next  step  was  to  correlate  the  Identified  system  objectives  with  the 
various  BIT/ETE  FOMs.  A  majority  of  work  for  this  step  was  accomplished 
during  the  first  step.  As  a  check  (to  make  sure  all  the  system  objectives  were 
identified  for  each  BIT/ETE  FOM),  each  FOM  was  checked  against  all  the  system 
objectives  identified  to  determine  If  there  were  any  other  existing  correlations. 
The  results  of  this  step  are  shown  in  matrix  form  in  Figure  6.  The  matrix  re¬ 
lates  the  BIT/ETE  FOMs  (tabulated  across  the  top  row  of  the  matrix)  with  the 


identified  system  objectives  (tabulated  down  the  left  side  of  the  matrix).  For 
brevlty«  the  system  objectives  have  been  denoted  by  numerals  and  abbreviated 
titles,  that  refer  to  die  numbered  system  objectives  tabulated  in  Table  17. 

The  final  step  in  the  guidelines  development  process  was  to  determine  which 
BIT/ETE  FOMs  were  the  beat  candidates  to  fulfill  eaoh  system  objective.  The 
method  used, was  a  ranking  of  the  BIT/ETE  FOMs  by  evaluation  scores.  Two 
methods  were  used  for  ranking  the  BIT/ETE  FOMs.  The  first  method  ranked 
the  FOMs  according  to  the  evaluation  rosults  of  Section  4.2  (FOM  suitability 
evaluation).  The  second  method  used  was  a  ranking  of  the  BIT/ETE  FOMs  accord¬ 
ing  to  their  suitability  towards  the  system  objective  it  pertaihB  to  (i.e. ,  the  FOM 
that  fit  the  objective  die  best  was  ranked  the  highest).  The  ranking  results  are 
tabulated  in  the  associated  boxes  of  Figure  0.  Eaoh  box  contains  three  pieces  of 
datat  1)  the  FOM  ranking  due  to  evaluation  results  of  Section  4,2,  2)  the  ranking 
due  to  the  FOM's  suitability  towards  the  system  objective,  and  3)  the  ranking  of 
the  average  of  R^  and  R2  (e.g. ,  the  rank  obtained  from  the  average’ of  the  two 
ranking  methods. )  For  oases  where  the  average  rank  was  the  same  for  more 
than  one  FOM,  the  FOM  with  the  higher  ranking  due  to  the  evaluation  results  of 
Section  4. 2  was  ranked  higher.  The  format  of  the  ranklngB  presented  in  Figure 
6  is  shown  in  Figure  7  below; 


Figure  7.  Ranking  Format 


Three  rankings  were  given  so  the  user  would  have  a  choioe  in  selecting  the 
ranking  soheme  dial  was  more  important  to  him.  The  rankings  are  very  similar 
and  the  use  of  any  ranking  should  not  significantly  affeot  the  group  of  FOMs 
selected. 

a.  1.2  Use  of  the  BIT/ETE  FOM  Specification  Guidelines  Matrix 

Selection  of  an  appropriate  set  of  BIT/ETE  FOMs  for  a  system  is  facilitated 
by  the  use  of  the  BIT/ETE  FOM  specification  guidelines  matrix  of  Figure  6. 
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Use  of  the  guide  tine?  matrix  includes  the  following  steps  (each  of  which  is  ex¬ 
panded  upon  in  the  following  paragraphs): 

1)  determine  which  system  objectives  are  desired  for  the  system  in 
question 

2)  for  the  system  objectives  selected,  determine  which  FOMs  are  the 

best  candidates  by  using  one  of  the  following  techniques: 
a)  based  on  the  B1T/ETE  FOM  evaluation  ranking 
bi  based  on  the  system  objective  rankings 

c)  based  on  the  average  ranking 

d)  based  on  the  user's  judgment 

3)  select  an  adequate  number  of  FOM  a 

Selection  of  the  System  Objectives:  This  task  consists  of  selecting,  out  of 
the  list  of  BIT/ETK  objectives,  those  that  pertain  to  the  objectives  desired  for 
the  system  in  question.  The  format  of  the  table  is  arranged  such  that  the  user 
scans  the  list  of  possible  objectives  in  Table  17  and  selects  the  objectivefst  most 
representative  of  those  desired. 

The  list  of  system  objectives  is  very  generalised  and  the  user  should  use 
discretion  when  applying  the  presented  methodology  for  selecting  BIT' ETE  FOMs, 
Only  the  necessary  system  objectives  should  be  selected. 

Selection  of  the  BIT  ETE  FOMs:  The  malm  is  set  up  such  that  when  a 
system  objective  is  selected,  the  user  scans  the  row  for  the  list  of  applicable 
FOMs.  Selection  of  *he  appropriate  FOM  can  be  accomplished  several  ways. 

It  is  recommended  that  the  BIT 'ETE  FOM  with  the  best  average  ranking  be  the 
specified  FOM.  However,  the  matrix  leaves  room  for  flexibility  by  listing  all 
the  applicable  FOMs  and  alternative  selection  processes,  such  as  rankings  by 
system  objective  suitability’,  and  rankings  by  the  BIT/ETI,'  FOM  evaluations  of 
Section  4. 

The  average  ranking  (BIT 'ETE  FOM  evaluation  ranking  and  the  system 
objective  ranking)  is  recommended  since  it  accounts  for  the  ability  of  a  FOM  to 
be  used  by  the  producer  and  it  accounts  for  the  suitability  of  the  BIT  ETE  FOM 
towards  the  consumer's  requirement  objective. 

Selection  of  an  Adequate  Number  of  FOMs;  In  the  process  of  Identifying 
the  desired  objectives  and  the  most  appropriate  BIT  F.TE  FOM  associated  with 
the  objective,  there  may  be  some  duplication  in  the  BIT  ETE  FOMs  selected 
(i.e.  ,  one  BIT  ETE  FOM  may  be  best  suited  for  mow  than  one  BIT,  ETE 
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objective).  The  problem  that  arises  is  determining  how  many  BIT/ETK  FOMa 
should  be  specified.  The  minimum  number  of  B1T/ETE  FOMa  that  should  be 
specified  are  the  unique  FOMs  that  have  been  Identified  by  the  desired  BIT/ FT  1. 
objectives  nnd  the  aforementioned  selection  process.  However,  if  the  user  so 
desires,  he  may  select  alternative  FOMs  (e.g. ,  the  FOMa  that  have  the  next 
highest  rsnidr^  or  any  other  POM  related  to  the  BIT  'ETE  objective)  for  the 
cases  when  there  arc  duplication. 
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\  high  HT  is  tvpicnlly  the  result  v»f  extensive 
HIT  IT:  caphHMt  >  .  The  capataUty  t  ypically 
results  in  n  high  iteyve  of  lUT'FTK  fault 
imitation,  therefore  a  low  T |.f  ’ 

A  high  1>1  is  typically  the  rcnult  of  extenai.e 
HIT  .’FT F  capability.  t  hus  a  hi|[h  *teg hi  of 
faults  are  inoIisIciI  oith  HIV  1  IT.  This  typi 
cally  results  In  a  loae-  MPSI.  rmptinctl  to  main 
tain  the  system  since  fault  iaola’ton  is  a  fune 
tied  of  it**'  HIT  K  i  ti . 

Without  any  implication*  to  tlie  Mootin'.  of  HIT 
TTl  there  is,  a  ma  MPSt  typically  results  in 


a  high  rn. 

T |^|  WTTR  Since  t  yj  I  in  element  of  MTTH  it  ahoula  he 

\  smaller,  Tvptcntly  MTTH  ixmaiHta  of  several 

\  s 

other  elemental  times  (o.g,.  dlsanscnibl y ,  re 
lUMMHiihty ,  checkout.  etc  1.  Therefore,  T  j 
f  ■  should  la*  considerably  lei«s  than  MTTH. 

MPHI  MTTM  Without  mtv  hwrlieathma  to  the  amount  of  »*<T 

I1  1 

rTT  ther*1  is,  a  tow  W|*$|.  typically  reaults  in 
a  high  MTTH  , 
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Not  only  should  the  selected  HIT/CTF  rO"  'j  be  quantitatively  consistent, 
they  should  also  be  qualitatively  or  intuitively  consistent.  For  example,  ft 
was  stated  previously  that  tht  specified  MTTR  must  be  greater  than  the 
specified  Tjq.  whin  aped  bed  together,  It  is  int-dtive  to  *ay  that  Tpj 
should  be  considerably  Issj  than  MY  TH  tor  sice  vers  it.  Thla  i»  obvioiw 
at  nee  MTTR  is  usually  comprised  of  fault  isolation  time,  disassembly  time, 
interchange  time,  reassembly  ti><« ,  and  checkout  time.  So  specific  con 
strain!*  can  be  determined  for  1X544 ■  that  are  qualitatively  related  How 
ever,  general  guide! nos  can  be  set  up  to  avoid  th-mc  types  of  inconsiaten 
dee.  Table  1  4  also  summarises  the  ronetrdnt*  on  FOV  «ped  float  ions  for 
qualitatively  related  POM*, 
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7.^  intfchation  or  iur  m  ro*u\\ir»i  n  smt-Kr  main  vAiNAiimn 

PttOGHAMS 

Thin  finHiiw  tilecunnr*  how  thn  profm*«xl  HIT  TTi  Kl>\Sn  anil  tholi- 
prtato  ilmvvmntmtiun  .insi  ctnahnin  uvhnivjut  f  va«  hi'  lnk<gr*U'<.t  into  futujv  maw- 
UlMMlitv  pn«(r«m  pliM,  A  revU’w  itf  thr  nUnUnl  miMtliublUt)  tan'«nm  *4 
Ml»i-STIV4Ti'  UUnUfU'n  tho  Rumr  U»K»  ju> 

ft.  l*tV|Mir  mjinUhiahilr.v  plan, 

h.  SVtfomi  nuintaitubUlh'  iUwNin. 

c.  I'hiun*  Iniwh  in  thi'  ilmiltsl  tium<i-iumv  ootuvyt  lf>'  iMuilix! 
miunlmwtv  ;ibn, 

KuLtMliN  nMinl.(liMlilih  nrMnrt  i, 

o.  IVrfnnr  UauW^-VT*. 

f,  I’mllrt  >uiint.UruJMht\  lur.tm  trr 

It.  *■>*«•  4  pm  I  m.tinUi«uMtli\  r  xjmI  n  .m-nt*  in  ftulfcMntnu'ti't 

i»ivl  \ ii»)nr  rju't  niH’lfii'iiiinns, 

h.  UiUitr4t«  “Ov  j  i|tni», 

i.  l.n  hmc»*, 

*.  *■  ftLthli.th  tl.itu  *oHcotiu«*»,  ui  \l>  nt*  ,uvl  oi«nivtiit'  .u'U»x»  amloni. 

k.  HvnumtiMli'  .uAimimirt  maud.uiuihiUti  rvxpt i n  nvnta . 

l.  Ihlilll.dlitl'illia  rit.ltllft  nil'll'. 

Tain.  -i*  mn how  nu  n  mmcr  tank  l.«  iiflittblr  U<  HOT  HTT  POM 
HHViflalniw  ,  THi'  (I'llinim;  Miiv.n  mi^thx  tllnounn  m  kAill  how  ftirK  nun'f 
Unk  oun  in*  i  rt it'll* nut' mini  foi  MIT  i  TP  PO\tn 

•  I’rvjMrr  nw,irrt.*»n.thi)lm  pri^ruM  >t*n  -  tin  maintuimhiiitv  |»n»* ram  plan 

In  a  »k',n\'rl|>tio*i  »4  how  Uu*  nmlnuA'i  ant.mln  to  onmliH  th**  nutNtakiMhlUti 
port  It  v\,  i4  i  n\ wte'tit  u»  Han't  tho  i^nvifUxt  mfuKvmi'HU.  Thu 

ooo»ifttft  ,4  i.lonUfv  ir*  Iih'  work  I'.’*  lx'  .«i'ivni|<liib«il  for  I'avlt  tank,  tho  Unr 
ph»n u*  *4  t-Av'h  Uni.,  oryanl. 'athm.il  ml  4>lmlnint ratl\ «  rvnp  mwkhtlHtt'n, 
ill.  wallow  .uni  ptv»IMIt*rt  Uvhttiifut'n  that  :ir.  to  In'  umxS,  t*U\ ...  AM  tho 
*fon*»M*'ntl«»)»>l  tank*  nMn  v.»«il\  lx:  to  HIT  r TP  POM  n|«4 villi' Attnna. 

•  lYrfnrni  itui«4.<liuliliil\  uiutnl"  -  Thin  tank  nwjulrtv  ,vn  aiuhnlM  <4  nviin- 
UintMlIU  iixftiiviwnln.  ThM  oowUin  <4  tnmnt.itli^  an.  I  nlUvatit*  th» 
manta  irvthtli  tv  miuiivnutiln  tlown  to  .ill  hrnotloml  lint'ln  whorv  ipplnMhh'. 
1  o*  HIT  ITP  POM  |  tvlfli'.ithxw*  mohx  n.dhul  nmint  !»'  iliti'liifHil  to 
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TABU:  20.  KNTKviRATMN  OF  BIT  1  TF  PRO*:  RAM  TASKS  VMTH  A 
MAJNTAISAHI1  II  ’i  I’fKK.RAM 

Wl.-STD-ITO  MainLiimlrilitv  Applicability  tu  1UT  KTIi 

Program  Tan  Its  |  Program  TvU 

Prepare  MainUinaMUlv  Pr\jgr.ini  ,  Direct  Applicability 

PUn 

IVrfnnn  MiM4iiuIiilitt  Analysis  K«h|uIivii  .t  imihml  of  .*U«iC4tIn»i  rwgilrc- 

?  menl*  (<■  lower  IcwU  (t  ««. ,  similar  to 
maintainability  allocations) 

« 

Prepare  Inputs  to  the  Detailed  \ktin«  D>.rcet 
Um&nce  Concept  ant  Dctailol 
MaintrntKr  PUn 


Establish  \LtinLunahiliiv  Dc*iigr  H.xgiinrs  jcuitlehnrw  lor  designing  1ST  ITi 

Criteria  ic .  £.  .  se«  SAVMAT  3SMith 


Perform  Design  TruU*Jn  IMivd  IippUcaMUtv 

IVwllct  UiintilniMlih  Ibnumtt  r  ltrt|iiin's  ^  proliduw  oi  analysis  t<vhiui|uc 
Values  t  refer  to  3ci4ion  *»,  !> 

inrorj'-'rsU  ami  I'nlnn>  Mamt.unabil-  llo|miVf  a  meth«*l  >4  rUnmliiy:  nxpmv- 
lt\  (UimvnicrU  in  ffciiiconlrici  .uvi  merits  to  toner  level*  <st  ,•  task  3  ube.ei 
Vmtur  HerllU-jUlow 

Wrjtntr  Othrt  tlfi  i*  iUxfum  *  i  nirthtyl  to  eontUne  Under  Imt 

parameter*  or  rrtpif  vvmcnU  t,*e«  task  n 

ot  -  ruuu  i  ! 


Participate  in  Design  Hn»\xn  Duvet  Vjplivabtlitv 


Establish  Date  Collection,  Vulvitis,  I  Di  n\i  AjipIk'&hlUtv 
Aifet  Corrective  Vtion  System 

Demon>t  rale  Vehlcvement  <4  \Uin~  lUnjuiret  i  tiemorvst  ration  Uvhnii|ur  «s*v 
tainabilitv  KviMirrmetit*  ’  Seeti >wi  •, ‘t 

Preparr  Maintainability  Sixtus  H«purt*|  Direct  Applicability 

.  ..  i  . 


allocate  the  B2T  ETF  requirement*  down  tn»  the  functional  levels.  Hie 
allocation  mettindkilngy  should  be  similar  to  existing  .ruintainabilltv  ailoc.i- 
tion  techni*#uc*  aiut  shtail  )  ,uMre«»  factors  such  ,»*  HI  talkin'  rates,  c«»m- 
p|extt>  (dlfltrttHyi  of  BP  FTP  implementation,  assm-lated  mstnUfwvblUtv  eon- 
side  rations,  and  coal  tu  tors  aherv  available.  TW-  miulrv*  an  uialvai* 
of  each  functional  level’s  BIT  FTF  capabilities  <l.e. ,  shat  de*rm>  ef  HI  f 
FTF  can  be  implemented  lor  each  (iir.dl«ml  level!  in  4»Wltlon  to  currently 
performsd  reliability  .vni  milntaln.ibilttv  juuIvm'k, 
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•  t'rvparv  'input*  to  tho  .Malta!  nuInti'miMH1  oonotv*  and  .total kv?  niUrto 

ruiK'o  plan  -  tin'  .Hlovotivo  of  this  task  is  to  prvnik'  t*<  ISo  .Vvol  • 

t^mn'rit  of  the  iM.tttal  m  ttntitunt'f1  conoovl  ind  nvn.infcnamipo  |>i  m,  Ttu* 
liHlhKk*i  an  »ruH  si*  of  th«*  avalUhto  mainlriMiii't*  rv<mtvM  (o.  *.  ,  pot'- 

lUtowmonUti  hi,  itu'illlios,  rto . i  and  tho  iv«t«irtv*  that  iro  mfulrtsl 
*o  achi«»v«*  tho  ,<»|nvltt»Nt  ouintatnahtlttY  oharaolfrtotlo'i,  Thi*  t.t  W  t * 
dlrvHUv  i^'tu-ilUc  lot  til’  l‘TI‘  spool  Cl  out  ions. 

•  LataHUh  malrtUinahilttN  ito*ij(i\  criteria  -  TV>  uhtMitt  M  tht*  tank  t*  to 
e«t*»4»*h  and  maintain  ro  ’tiytuinaMUtH  f»n‘>i*n  entcrit.  (•stuMtshlnir  On' 
ifaMlL'it  orttrrtn  c  nsist*  of  defining  the  mututumditlih  fcatu re*«  to  hr  ln~ 
chato**  In  On-  .h-mgrt,  In  order  to  iwvl  tho  spool  find  m  lint  .tin  tlti  lit  t  rt'^juliv* 
monts.  Some  oyaiivU-  iW  l.vigu  ipihtrline*  an1  uov»‘s*lh*llt\  ,  lost  points, 
Intorvh.u.godslitH .  fail  solo  h'.ilmn,  cto. . , ,  Tld*  task  is  dtrvotto  audi¬ 
table  to  Hit  l  'll  It'll  <t|t<Mlit-tlki>n  sine*-  nuIntunttMlilt  design  guide 
llmt  can  sl*«<  h>'  ntlaMmlhxl  (■  r  HI  I  I  i  t  H*M»  NAIH-TH  -  “s  >?«.  * 
IVtUgu  i.unlc  lor  Muill  In  Vi  »l,  t«  a  g.Hsl  source  of  PIT  ‘  I  *  di-sign  lc« 
turv>  amt  ajH'roa.  h>  -, 

•  IH’rform  do*  is’i  trute.41*  -  Dn*  u  k  mpur  -  the  rt'.mtlv  *\*  oni  d.vunuix- 
twtiou  of  ant  thing.  *  tK.it  are  IV  result  ol  dc*i£n  tradeoffs.  VStki  d*.’ 
poit.iuiu'  to  IWT  IT!  tnalvnit, 

•  I’lVilit'l  m.tln(.llMliitilt  p  tctnvtoi  -  !*»»-.  u*k  topoi.'  the  prediction  of 

the  iguiit  native  ni.tintum  Midi  lot  the  planned  design  nortfltf- 

u  rot  ion..  Vor  ItT  VTV  parameter*  tf’OMsi,  the  <fu.uvtto.dlHo  v .tint's  euv 
ln«  pmllrtnl  using  the  treHittifui *  dtsouwscd  trs  Section  r«,  I  of  thlu  report. 
AikHttinvU  pnsdlotlnh  rocth>d.4oiric*  ntisl  lo  In*  ■  ’ewtt'ped  for  Minn*  KOMt 
to. Bn  t  l‘  V,  ;c*d  Kf  XU. 

•  imvnimnfr  tiki  »,nloi,'o  i»uiniaiiMlSl't\  mjoiit  on  nts  m  sultconl  motor  utsl 

vendor  *|* » vtf  Um tl» »»%?*  -  rhi*  task  oniaists  >4  m»,  tju  u'tiUUt  r  nt.on- 

taln.«ltillt\  tv«mltvnM*nt»,  on  tt'mlt't  ml  «uln>>ntr.U'tov  itoios . 

This  c.ui  ttso  In*  :u-*>*nt|'llslitkl  lo»  Hi  l  1  1  1  itHium'iuonts  t»\  vt -*mtt  th<' 
itlliH'atitnis  tt*fhitii|w»'it  ilist'vsittsi  vrt'vtoush  , 

•  !titrgr»t«'  itOtt'f  itt'Uut  -  'Ittis  ttsk  ontststn  «*f  .Momtintivt:  tin  oh  or  ill 

quantit  itino  ttt  tihLtlihtl4tltv  |k»r  tmoti  iw  vltioh  iin‘lu>lt‘«  *j**vo»  »m»ont  ftimisInHl 
tMfuiititu'ivl  or  .utwa'Uto  unlrtnior  oviutpnn'nl .  This  is  \l*o  t|yli- 

u.iWt*  to  i|U  u»tH;lt»vo  IUT  1*1 1  |VM‘;imotor«  jhi-i'h  uIohI  tl'.it  i  nuHtital  ovist*  lo 


combine  tower  indenture  level  BIT  KTK  parameter*  into  higher  level". 

Tills  can  In*  accomplished  In  the  revers*’  pryx'«ts  of  the  allocation*  tech¬ 
nique  or  bv  using  the  analysis  methods  that  are  prcsmtcsi  In  Section  f>.  1. 

•  Participate  in  design  rev  leas  -  This  task  consists  of  reviewing  the  analy¬ 
sis  data  to  assure  accomplishment  id  the  specified  requirements  (n  appro¬ 
priate  stages  of  system  equipment  development.,  This  task  is  directly 
applicable  to  the  periodic  review  of  HiT  ITI  characteristics. 

•  t.siablish  data  collection,  analysis,  and  corrective  action  r>  stem  ~  This 
task  consists  id  establishing  a  dita  collection  s\  stem  such  thsl  the  data 
uUaimsi  tan  be  usjd  to  provide  further  evidence  that  the  specified  require¬ 
ment#  haw  or  have  not  been  met.  A  data  collection  system  estahllshol 
for  the  purpose  of  collecting  maintainability  data  can  be  dt rvetlv  expanded 
t«>  inchi'k'  the  collection  and  iviivs  of  ItIT  fTI  ilat.i. 

•  nemisistrau  avhicvctm  nt  of  maintainability  requirements  -  This  task 
consists  of  sUPsticafiv  d.»MK>iu»trating  the  achieved  value#  of  the  maintain- 
ability  par.unete rs  usiig;  the  demons t n»th*e  techniques  provided  in  M11-- 
Ml'-lilii  The  demons! cali»*n  of  the  „ju imitative  IUT  I  Tl  pa.'iineier*  can 
be  aci  imph*ru<d  uainj;  Jhe  techniques  discussed  in  Section  o.  J  of  this  tv- 
port,  For  many  of  the  HIT  ITT  KOXis,  the’  sq«eetfied  maintainability  lostco 
c  m  be  eqiaiyd.vl  or  the  .!  ita  u*nl  di-pdl'  lo  evaluate  fUT  TTf 

require!  wnts. 

•  Prepare  inamtaitwbiUty  *Uius  report*  ~  At  Interval#  determined  or  .q>- 
pnood  by  the  p  recurring  activity  ,  statu*  rty»ort*  shall  U*  submitted  sh  ch 
provide  the  current  assessment#  of  the  nuintudnabiUV  parameter*  and  on 
other  information  pertaining  to  ihc  maintcn.uu't  of  the  system.  This  can 
be  directly  ipttliod  to  SIT  H  i  iMninu'im:  anil  l!T  KIT  related 
information. 


8.  0  t’ON<  •  t ,  U8IOKS/HE  COM  M  E  NI>A  TIONS 


The  program  objectives  identified  in  Section  1  of  this  document  have  been 

accomplished.  The  res  u  I  Is  obtained  thru  this  study  provide  the  necessary  infor¬ 
mation  to  adequately  specify,  analyse,  and  demonstrate  the  BIT/ETE  capabilities 
contained  in  a  aystem/equipment,  The  methodologies  developed  are  compatible 
with  existing  maintainability  program  techniques  ami  allow  BIT  'KTK  requirements 
to  be  easily  integrated  Into  standard  maintainability  programs. 

Specific  results  of  this  study  include: 

1)  The  dam  collection  task  showed  that  the  BIT  KTK  FOMs  currently 
spcclilcd  'used  do  not  cover  all  aspects  of  BIT/ETK  capability  and  are 
ambiguous  and  Inconsistent.  This  study  establishes  firm  definitions  of 
the  BIT/KTF.  FOMs  examined.  It  la  recommended  that  the  established 
definition  or  variation  thereof  be  Incorporated  in'*-»  a  military  standard, 
such  its  MU.-STD-7U1. 

2)  Methodologies  h  ive  been  defined  to  adequately  analv/e  end  demonstrate 
system  equipment  HIT  HT  capabilities.  Analysis  methods  consist 

of  rate  dependent  techniques,  time  dependent  techniques,  combined  rate 
and  time  dependent  techniques,  mat  current  maintainability  analysis 
techniques.  OcmonHtraiion  methodn  consist  of  statistical  demonstrations 
using  a  binomial  ilisiributton,  a  multinomial  distribution,  and  current 
maintainability  demonstration  techniques. 

*.V<  A  technique  was  developed  to  methodically  select  the  MIT  KTK  FOMs 
that  should  appear  In  a  system  ''equipment  specification  based  on  the 
desired  system  equipment  objectives.  Along  with  the  developed  teclv- 
tuque, guidelines  have  been  presented  to  aid  the  user  in  specifying 
numerical  values  for  the  BIT  Hi  FOMs  and  constraints  that  must  be 
followed  when  specifying  related  BIT  FTF  FOMs 

41  This  study  provides  the  necessary  tools  required  to  integrate  the  do- 
fined  BIT/KTK  FOMs  into  standard  maintainability  programs. 

The  results  of  this  study  prov'de  the  foundation  for  the  consistent  specifica¬ 
tion  and  verification  of  effective  BIT/ 1'  i'K  F'OMs.  Knhanceinent  of  the  prov idl'd 
cspabilities  Is  tccommeodcd  by  further  studies  1 1  the  follow  ing  areas. 
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I*  False  Alarms  -  Investigate  and  analyse  t!  .*  causes  of  false  alarms, 
specifically  drill  and  notar  related  effects.  Develop  nicthodolojjics for 
Ihr  analyal*  ol  false  alnrms  Inchtdln#  (rcquiwv  of  occurrence  amt 
far  lorn  affecting  their  occurrence,  Develop  leelnikpic*  for  physically 
demonstrating  false  alarm*  lmTntllit£  tndiiccim  ■>< ,  simulation  and  or 
modeling. 

2)  Automated  Circuit  Analyst  •  Investigate  (he  current  methmlologtcs 
available  for  automated  (Computerlrcdl  circuit  analysis.  Compile  the 
available  tccfwikjucs  and  develop  new  techniques  ts  noeeKKcirv  to  |*e in¬ 
form  automated  analysis  of  circuits  relative  to  out- of- tolerance  coral  I - 
lions,  transient  conditions,  failure  modes  and  effects,  ami  umk'U'CUNl 
fa  Huron.  These  analyses  could  be  directly  a  tribal  to  tire  evaluation  of 
lirr  ^Tf  H>M*  such  as  FFD,  FFA,  ami  1  FSI. 

.tt  Coat  Tradeoffs  Thr*  evaluation  of  HIT  I'l'f  IHMt»  presented  In 

Section  l  and  the  guldt'liitc  for  thr'  selection  ol  speelfieaUon  of  these 
FOM*  presented  In  Section  ('•  have  purposely  avoided  coat  relationships, 
The  subject  of  oust  la  a  significant  factor  In  the  selection  and  implemen¬ 
tation  of  IIIT^FTK,  la<t  thr'  ansivsl*  of  cost  Is  as  Involved  as  da'  entire 
study  of  linVTTK,  first,  cost  muat  he  deftryed  (c.g*.  thmlgn  cost, 
recurring  product  k>n  costs,  nonrecurrent  costs,  life  cycle  coal,  docu- 
mentatlon  coat,  aast'claled  nialntcnancc  mnopowet  costs,  etc.  l.  After 
coat  la  defined  (amt  lhr>  def  inti  Iona  are  as  varied  as  the'  systr  nr  for  which 
the  definitions  are  made),  It  must  then  be*  related  to  HIT  *‘Tf  chat ac- 
torlalios.  For  example,  fault  Isolation  resolution  to  a  single  HI  90  per¬ 
cent  of  thr>  time  for  one  million  dollar*  Irr  design  me  t.  Is  equivalent  to 
isolation  to  *  single  lit  -0  percent  of  thr  time  for  half  a  million  rtolUrs. 
Arch  an  equivalent  must  consider  all  related  factors  such  as  mission 
criticality,  spares  availability,  personnel  skill  levels.  support  equip 
nivttt,  He.  .  Thr'  development  of  typical  H  I  f  I  F  FOM  anti  eost  rein 
tionships  anri  quantitative  interrelationships  between  HIT  I  IT  f  OMs 
would  provide  «  very  usefkl  tr;nk,off  t<  hmquc  for  the  selection  anri 
specification  of  H. »Ms. 
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APPFNWX  A 


A ■  1  BINOMIAL  DPT RlBimON  TABLES  FOR  STATISTICAL 
DEMONSTRATION  py  FFD.  FF1  AND  ELF1 


The  following  table*  provide  the  probabilities  of  passing  She  dtawmlrn 
Mon  teat  a  described  in  Section  9.2.1.  The  'Miles  presented  err  for  a  pre¬ 
sent  stive  sample  of  test  statistics.  That  is.  the  tables  presented  are  tor 
samples  sites  Of  n  =  20.  29.  30.  39.  40.  45.  SO.  60.  70.  10.  M  and  100.  and 
pass /fail  criterion  t tor  each  n)  of  k  -  O.OOn.  6.60n+l,  Q,00n*2.  ...  n  l , 
I'sage  of  those  tables  are  described  in  Section  5.2.1  of  this  report  end  in 
.Section  A. 2  of  this  appendix. 

The  values  presented  in  the  tables  were  cxwmputed  using  the  following 


vipt  £  pV***‘ 

I  k*  t 

where; 

n  -  aample  site, 
k  -  pass  Tail  criterion. 

p  *  the  true  value  of  the  POM  being  demonstrated  tof  the  type  discussed 
in  Section  5.2.1.11. 

>(p)  c  the  probability  of  paasing  the  desainatratlon  test  if  p  Is  thr  true 
value  of  the  FQM  being  dmiittlral«l  (of  the  type  discussed  in 
Section  9.2.1.11. 

The  producer  risk  •  is  given  by  i  I  >(pt>)  where  ^  is  the  design  goal, 
and  the  consumer  risk  r  is  given  by  e  t(pjl  where  p(  is  the  minimum 
acceptable. 

A. 3  Step^l^Sley  Procedure  lor  llesignin^  s  Demons' raliun  Teat  Pal  tig 

Case  t:  n.  P|,  r  known,  k  and  pfl  must  lie  determined  (assuming  a  reason 
able  value  for  i  .  say  i  t*  > . 

Step  I  Example  the  tables  in  Appendix  A  for  those  cases  where  sample 
sl*e  U  n. 

8t®|>  2  Mud  the  value  of  k  (i.o,  ,  pass, Tail  criterion  1  for  which  the  table 
entry  (i.t  . .  "probability  of  passing  the  lest  >  corresponding  to 
Pj  (i.e.  ,  pj  as  the  trim  value  of  p>  la  ctnseet  to  thr  desired 
This  I*  the  value  of  k  re<|uined  and  the  table  entry  correspond 
ing  to  p|  la  the  exact  value  of  i», 

n 


Step  a  For  the  «  given  aivl  k  (it  Step  2,  search  the  table 

uortvapumling  U*  ample  alie  it  ami  pan*  Tail  criterion  h  for  the 
entry  (i.e.  ,  "probability  of  pawning  the  teat")  e<|U«l  (tntcrpola 
turn  may  be  neceowary )  to  t  *»*  The  i»rrct»tMMHliiig  true  value 
of  |*  in  (hr  required  Pp  (for  the  wvunuxl  .1  1. 
mm  1:  p,  Pp  » ,  |»g  ill  kumn,  n  Niui  k  muni  be  tU'l^rmiiutl . 


Step  I  Compute  a  flmt  approximation  t<>  it  iwtiiK  ix|uiitftiit  S.3.I.  I.K. 

Step  2  Compute  a  (llwl  approximation  to  k  uni  up  equation  S.I.I.I.*, 

S|«p  3  Fitter  the  table  In  Appendix  A  wrrv*i*>iKting  to  a  sample  mlw  of 
n  ami  p ana  Tail  criterion  of  k.  Krwi  the  exact  value  of  t  from 
the  table  entry  <|.e..  "pivbaWity  t*f  paaalng  teat“l  (er'MpiwI 
ing  to  |  (l.e.  .  P|  »<  the  true  value  nf  pi.  Compute  tl*e  exait 
value  of  .i  iw  one  min  via  (In  table  entry  o>rri<«|iim<ltii|"  to  |>l( 
(l.e..,  |'y  aa  the  true  value  of  p>. 

Step  4  Compare  Me  exact  v  alued  of  i.  v  with  the  original  valwrx  of  i  . 

t:.  If  they  are  chute  enough  (In-re,  Vk**»e  enough”  h  itetenminevt 
t»v  uaer  preference).  the  teat  i*  determined  by  uamg  the  value* 
of  u  amt  k  determined  in  Step*  {  anti  i.  If  the  exact  value*  of 

o  .  i>  arv  not  ekwe  eiaiugh  to  tl»e  original  vatu**  of  » ,  r  .  then 

vary  k  ami /or  n  (i:auat‘,y  inereaar  n)  alight  ly  and  roped  Step  3, 
Continue  until  the  «*•(  \  aluo*  of  >  ,  r  are  vWe  enough  to  the 
original  value*  of  t  .  f  . 
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APPENDIX  ft  -  mi 04  DCMOWnHATION  TEffT  PLAN  EXAMP  LIS 
TaMtos  ft-1  Arxft  ft-*  ooMaln  wapllw  of  lift*  atatiatfoat  daaaottftraftkta 
Mfcatqaa  daMrtfcad  far  PIIML)  la  ftacttoai  t.S.S.  A  (hwcrtpUo*  w  how  •»  ftM 
pkt*  mf  tewlopli  aad  lo  on  Km  J-*w«rHm  M  plw  t*  wWwd 
iBftMUM  t.a.i. 


TABU  B-I.  1XAMFL1  l  Of  A  DEMONSTRATION  TEST  PLAN  BOB  HBOj 


Daalgn  0o«l  Value *; 


HR(L1)  «  0.1000 
HR  (LI)  -  0.9600 
FIR(LI)  -  1.0000 

NfllaiM  Acoaptable  Valuaa: 

fir(li)  •  o.oooo 

FIR(LS)  -  0. 9300 
HR(U)  *  1.090 


Fraapodflad  Conaiaaar  Mak  (Bala) ; 
CoafAciant  in  Taat  Statistic, 

0.05710 


Saatpla  Site 

Exact  Bata 

50 

0.0001 

00 

0. 10U 

70 

0.0150 

00 

0.0104 

00 

0.1011 

too 

0.0040 

110 

0.1011 

ISO 

0.0050 

iso 

0.0007 

140 

0.0*14 

100 

0.0971 

0.1000 


-0.47000 

-0.39047 

Exact  Alpha 

Paaa/Fdl  Value 

0. 0353 

0.0943 

0.5416 

0.0130 

0.5350 

0.9819 

0.5019 

0. 9800 

0.4409 

0. 0110 

0.4359 

0.0000 

0.5700 

0.0700 

0.5070 

*  0.0500 

0.3303 

0.0314 

0.3300 

0.7004 

0.3600 

0,7511 

145 


TABLE  b-t.  EXAMPLE  l  OF  A  P1HOH8T  RATION  TEST  FLAH  W>  ntlL) 


Dealfn  Qotl  VtluM : 

FlB(Ll)  *  9. MM 
FI*  (LI)  »  O  MM 
F1R(L3)  «  1.0009 

MlniMUM  Acceptable  Values: 

FlR(Ll)  -  O.OMO 
FIR  (LI)  *  0.HN 
F1R(L3)  *  l .Mi 

Fveepedfled  Cunauatf  Wak  (Beta):  0.2990 
Coeffidants  in  Teat  Statistic : 


0.06710 

<0.47000 

0.33047 

Saapk  Site 

Exact  Bata 

Exact  Alpha 

Pmm/F*1  Value 

M 

0.1000 

0.4101 

0.3003 

M 

0.2030 

0.3033 

0.3300 

to 

0.215? 

0.3110 

0.3007 

•0 

0.1000 

0.3041 

0.2005 

w 

0.1034 

0.2004 

0.1074 

IM 

0.3001 

0.2337 

0.1341 

110 

0.1130 

0.2000 

0.0073 

IM 

0.1001 

0.2064 

0.0374 

139 

0. 1000 

0.1003 

-0.0251 

140 

0.3030 

0.1500 

-0.0001 

too 

0.2037 

0.1422 

<0.1572 
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TAB -JR  8-3.  EXAMPLE  3  OF  A  DEMONSTRATION  TEST  PLAN  FOR  FtR(L) 

Design  Goal  ViIum: 

F(R(L1)  *  0. 8600 
IFIR(LS)  «  t.NM 
FIR(L3)  -  1.004 

Minimum  Accept abk  Values: 

FIR  (LI)  *  0.0000 
F1R(L3)  »  0.0400 
FIR  (1,3)  »  1.0000 


PraapectSed  Consumer  lUak  (Beta):  0.1000 


Coefficients 

in  Teal  Statistic 

0.05407 

-0.11750 

-1.00041 

Sample  31m 

Exact  Beta 

Exact  Alpha 

Pass  /Fail  Value 

50 

0.0072 

0.4341 

1.0050 

to 

0.0040 

0.3777 

1.0302 

70 

0.0000 

0.3377 

0.0731 

•0 

0.0031 

0.2757 

0.0017 

00 

0.0100 

0.2432 

0.7070 

100 

0.0012 

0.2025 

0.4033 

no 

0.0»4 

0.1530 

0.5700 

no 

0.0076 

0.1340 

0.450T 

130 

0.0047 

0.1151 

0.3300 

140 

0.0032 

0.0007 

0.1044 

150 

0.0042 

0.0812 

0.0574 

i  i 
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TABLE  B-4.  EXAMPLE  4  OF  A  DEMONSTRATION  TEST  FLAW  FOB  PIK(L) 

Design  Goal  Values: 

FIR(Ll)  »  O.NN 
F1R(L2)  *  0. 9900 
FIR(L3)  -  1.000 

Mlniaun  Acceptable  Values: 

FIR(Ll)  «  0.0000 
PIR(LS)  *  0.0400 
F1R(L3)  «  1.0000 


Prespedfled  Oonsumer  Risk  (Beta): 

Coefficients  in  Teat  Statistic. 

0.2000 

0.05407 

-0.20760 

•1.00001 

Sample  Slav 

Exact  Beta 

Exact  Alpha 

Pace  /Fail  Value 

50 

0.1S00 

0.2130 

0.2157 

00 

0.2110 

0.1T00 

0.0050 

TO 

0.2X01 

0.1350 

-00500 

00 

0.2104 

0.1007 

-0.2001 

00 

0.2020 

0.0070 

-0.3400 

100 

0. 1059 

0.0014 

-0.5374 

110 

0.1941 

0.0050 

-0.7100 

ISO 

0. 20)4 

0.0400 

-1. 0005 

1)0 

0.20X4 

0.0302 

-1.7033 

140 

0.2035 

0.0300 

-1.0504 

ISO 

0.2040 

0.024) 

-1.4100 

140 


.. 


TABU  B-0.  EXAMPLE  6  OF  A  DEMONSTRATION  TMT  PLAN  FOR  PIR(L) 
Dwlffn  Ooal  Vahm: 

n«<U)  -  O.MM 

nR(U)  -  0. MOO 
flK(LS)  »  1.0400 

MMim  Acceptable  Values: 

FlA(Ll)  -  0.1000 

pih<ls)  «  o.rno 
nit(U)  -  i.oooo 


PmpMlStd  0»muMr  !Uak  (Beta) .  0.1000 


CMflldtnU 

in  Teat  Statistic: 

0.13909 

-0. OS 992 

-0.22314 

laapto  81m 

Exact  Bata 

Exact  Alpha 

P wax /Fail  V 

90 

0.0909 

0.3057 

0.9226 

•0 

0.0975 

0.2970 

0.7319 

TO 

0.0919 

0.2373 

0.9229 

M 

0.1099 

0.1992 

0.4994 

M 

0.1019 

0.1434 

0.3921 

100 

0.0969 

0.1229 

0.2109 

110 

0.0910 

0. 1001 

0.0909 

190 

0.1049 

0.0726 

-0.1020 

190 

0.0902 

0.0920 

-0.3712 

140 

0.0991 

0.0039 

-0.4492 

100 

0.1019 

0.0397 

-0.9264 
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APPENDIX  C  -  INDUSTRY  SimVKY.DA.rA 

The  tables  Uatcd  on  tin-  follow tng  inaifc*  the  data  iwrtwd  through 

thr  tnchMIrv  survey  (see  Section  4.4*.  The  table*  * r*i‘  organ!  *e«i  In  POM*  In  the 

fallowing  orden 

Tiblr  C-t.  *1  FI)  -  pvrtnit  tvS  fault  *  detect*  d  by  BtY.'KTK  fanslogou*  to  ITDi 
Ttbk  C-B.  FAR/ FBI  -  bbv  ilxmt  mu-  and.  or  false  statu#  iiuhonuon* 
(analogous  u>  FFA  and  FFSlt 

TabU*  C*S.  FIAL-  bull  UuhUun  ambiguity  level  («n»loiiiuw  W>  FIR<F|> 

Table  C-4.  TK1  -  percent  of  (teult  isolation  with  MIT  FTF  (Biwtogou*  to  FK’> 
Table  C-5.  S\S  MTBF  -  system  MTBF 
Table*  C-6.  MPM.  -  maintenance  personnel  skill  level 
Table  C-7.  IV  E  Rcl  -  HIT  FTF  reliability  {analogous  to  MTUI'  y 
Table  C-S.  B  'K  'A  T  &  VOI.  -  BIT  FTF  weight  'volume  tsnalogou*  U) 
phviU-nl  imuwt  Kilt  FVUMsi 
The  abbrrv  at  ion*  and  used  lit  tlu*  tables  *r<; 

S>  COMPANY  -  denote#  the  company  that  the  survey  «ih  rvcoIV'*d  from. 

Companies  a  err  denoted  by  nuw-vsl*  only. 

Si  FtjVIP  -  denotes  the  equipment  t\  vh-  the  !KOM*  haw  been  experienced  on 

AV  -  avtontr  equipment 
Ml  -  mlwilfH 

tiSU>-  ground  electronic  cqt<l|'mciil 
Mil*  -  shipboard  equipment 
OTM  -  other 

Si  EXPKRIKKCE  -  three  column*  denote  Uu-  surveyed  company  ‘s  espenence 
alth  the  POM.  A  check  mark  denotes  that  they  have  had  experience  with 
the  FOM  In  ottc  of  the  (ulU  litf  areas,- 

SPEC  -  thru  spec  If  tor  5  Km* 

ANA  -  thru  analyst* 

DEMO  -  thru  demonstration 

4)  SUITABILITY  -  there  column*  denote  the  aroma  given  for  earh  Ft >M* 

suitability  factor.  The*  suitability  factor*  an*: 

TRANS  -  trtnda  lability 
TRACK  -  trackabtUty 
DEMO  -  demonstrsbillty 
AMB1G  -  ambiguity 
C.F.N  -  generality 
COST  CtMII 

AVF  -  the  average  score 
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TABLE  C-i.  VOX  •  %  TO  fOoMHMMd) 


a* 


TABLE  C-8.  rOM:  FAH/FSI  (CovttMffd) 
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TABLE  C-4.  TOM:  %  f! 


1a*1*E  C-5,  FOM:  SYft  HTBF 


USE  Q£A£BQiai8g  AM?  SXMBQlfi 


Ac  rooy  m/%mbol 

DsflUinu 

A 

availability 

AB/£ 

|  BIT/ETE  availability 

BIT 

|  built-in  Vet-" 

BH/ETE 

j  buM-lo  test  and/or  oxttrasl  test  equipment 

b/e 

built -Is  tost  and/or  external  test  equipment 

EtflR 

erroneous  (suit  isolation  rate 

ETE 

i  extern*]  test  sgutpawt 

fb 

|  frequency  of  14T/ETE  executions 

FEF1 

!  frsetloa  of  erroneous  fault  isolation  results 

FFA 

|  fraction  of  false  alarms 

pro 

|  fraction  of  faults  detected  bj  BIT/ETE 

ffda 

!  fraction  of  all  faults  detected  by  BIT/EUE 

ppdd 

fraction  of  detectable  faults  detected  by  blT/E  EE 

Pfl 

fraction  of  faults  isolated  by  BIT/ETE 

ppp 

fraction  of  false  pulls 

PI’S 

|  fraction  of  falsa  status  indications 

Fl&lL) 

|  fault  inolaUoa  resolution 

PUAB 

1  fraction  of  memory  allocated  for  BIT/  ETE 

POM 

>  figure  of  merit 

PPK 

false  pull  rate 

GIOEP 

<  Government  Industry  Data  Exchange  Program 

LRU 

j  Use  replaceable  unit 

mct  e 

repel  r  time  of  the  itb  system  component 

“CTk 

i  repair  Ume  for  the  kth  BIT/ETE  componeat 

MPftL 

[  maintenance  personnel  skill  level 

MTBP 

:  mean- time- between- fall  urea 

MTRPb/E 

|  BIT/ETE  reliability 

MTTR 

mean- time -to -repair 

MTTRb/E 

|  BIT/ETE  maintainability 

the  quantity  of  BIT/ETE  last  routines 

*b/e 

|  quantity  of  BIT/ETE  tardw^r*  compose** 

*C 

!  numbt  r  of  component*  In  s  system 

*R 

I  number  of  Us  is  the  system 

LIST  OF  ACRONYMS  AND  SYMBO_LS  (CouUnued) 


Acronym/  Symbol 

Definition 

Qbdf 

quantity  of  faults  detected  by  BIT/ETE 

Qbif 

quantity  of  BIT/ETE  indicated  failures  (QbIF  CQFA  +  Qbdf^ 

Qefir 

quantity  of  erroneous  fault  isolation  results 

Qf 

quantity  of  all  faults 

Qfa 

quantity  of  false  alarms 

Qfd 

quantity  of  faults  detected  by  any  means 

Qfir 

quantity  ol  fault  1  eolation  results 

Qgri 

quantity  of  good  Rls  removed 

Qib 

quantity  of  faults  isolated  with  BIT/ETE  (Qib  <  QbDF) 

Qil 

quantity  of  detected  faults  isolatod  to  <L  Rls  by  BIT/ETE 

Qnbdf 

quantity  of  faults  that  are  detectable  but  not  by  BIT/ETE 

(Qnbdf  =  Qfd  -  Qbdf) 

Qrr 

« 

quantity  of  Rls  removed 

Qud 

quantity  of  undetected  failures  (QUD  *  Qf  -  QFD) 

HI 

replaceable  Item 

RMA 

reliabilitv/maintainability/availabillty 

S 

the  average  fault  isolation  group  size 

tb 

mean  BIT/ETE  running  time 

Tb, 

running  time  of  the  ith  BIT/ETE  test  routine 

tFD 

moan  fault  detection  time 

tFI 

mean  fault  isolation  time 

Tt 

test  *  ho  roughness 

Of 

produce  risk 

0 

consumor  risk 

y 

the  rate  at  which  transients  occur 

6 

the  rate  at  which  out-of-tolerance  conditions  will  occur 

XUFj 

the  failure  rate  of  the  Jth  BIT/ETE  component  that  results  in  a 
falso  alarm 

XB/E 

failure  rate  of  the  BIT/ETE  hardware 

XB/Ek 

failure  rate  of  the  kth  BIT/ETE  hardware  component 

failure  rate  of  the  ith  component  that  is  detectable  by  BIT/ETE 

failure  rate  of  the  Ith  component  associated  with  the  denominator 
of  the  defined  FOM 

LIST  OF  ACRONYMS  AND  SYMBOLS  (Continued) 


Acronym/Symbol 

Definition 

xEFIi 

failure  rate  of  the  ith  component  that  results  in  an  erroneous  fault 
isolation  result 

XIi 

failure  rate  of  the  ith  component  that  is  isolatahle  by  RIT/ETE 

M 

failure  rate  of  the  ith  component 

M 

failure  rate  of  the  Jth  R1  or  unit 

XLKj 

failure  rate  of  the  Jth  R1  associated  with  the  L^  fault  isolation 
resolution  level 

*ni 

failure  rate  of  the  ith  component  associated  with  the  numerator  of 
the  defined  FOM 

XT', 

*i  i 

failure  rate  of  the  1th  component  that  is  tested  by  BIT/ETE 

XUDi 

failure  rate  of  the  ith  component  that  is  undetectable  by  any  means 

^UDj 

failure  rate  of  the  jth  RI  or  unit  that  is  undetectable  by  any  means 
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